[==========] Running 56 tests from 4 test suites.
[----------] Global test environment set-up.
[----------] 30 tests from SecurityITest
[ RUN ] SecurityITest.TestAuthorizationOnListTablets
Loading random data
Initializing database '/tmp/dist-test-taskUci3DU/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781966829091300-388-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 20 14:47:09 dist-test-slave-q9j6 krb5kdc[395](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 20 14:47:09 dist-test-slave-q9j6 krb5kdc[395](info): set up 2 sockets
Jun 20 14:47:09 dist-test-slave-q9j6 krb5kdc[395](info): commencing operation
krb5kdc: starting...
WARNING: Logging before InitGoogleLogging() is written to STDERR
W20260620 14:47:10.656637 388 mini_kdc.cc:121] Time spent starting KDC: real 1.472s user 0.000s sys 0.017s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskUci3DU/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781966829091300-388-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskUci3DU/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781966829091300-388-0/krb5kdc/test-user.keytab.
Jun 20 14:47:10 dist-test-slave-q9j6 krb5kdc[395](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781966830, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-20T14:47:10Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T14:47:10Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.97.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskUci3DU/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781966829091300-388-0/krb5kdc/kudu_127.0.97.62.keytab.
Entry for principal kudu/127.0.97.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskUci3DU/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781966829091300-388-0/krb5kdc/kudu_127.0.97.62.keytab.
I20260620 14:47:10.827666 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskUci3DU/build/asan/bin/kudu
/tmp/dist-test-taskUci3DU/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskUci3DU/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781966829091300-388-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskUci3DU/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781966829091300-388-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskUci3DU/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781966829091300-388-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskUci3DU/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781966829091300-388-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:35831
--webserver_interface=127.0.97.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskUci3DU/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781966829091300-388-0/krb5kdc/kudu_127.0.97.62.keytab
--principal=kudu/127.0.97.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:43677
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.97.62:35831
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskUci3DU/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781966829091300-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskUci3DU/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781966829091300-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskUci3DU/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781966829091300-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 14:47:11.164325 408 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 14:47:11.164726 408 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 14:47:11.164896 408 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 14:47:11.172714 408 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 14:47:11.172880 408 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 14:47:11.172987 408 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 14:47:11.173091 408 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 14:47:11.173179 408 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 14:47:11.183173 408 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:43677
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskUci3DU/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781966829091300-388-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskUci3DU/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781966829091300-388-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.97.62:35831
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskUci3DU/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781966829091300-388-0/krb5kdc/kudu_127.0.97.62.keytab
--principal=kudu/127.0.97.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:35831
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskUci3DU/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781966829091300-388-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.97.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskUci3DU/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781966829091300-388-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 13:41:20 UTC on ccycloud.quasar-vdagoi.root.comops.site
build id 4457462
ASAN enabled
I20260620 14:47:11.184685 408 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 14:47:11.186087 408 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 14:47:11.199088 413 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 14:47:11.199167 416 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 14:47:11.199160 414 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 14:47:11.199899 408 server_base.cc:1034] running on GCE node
I20260620 14:47:11.200917 408 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 14:47:11.202472 408 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 14:47:11.203709 408 hybrid_clock.cc:648] HybridClock initialized: now 1781966831203634 us; error 69 us; skew 500 ppm
I20260620 14:47:11.204088 408 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 14:47:11 dist-test-slave-q9j6 krb5kdc[395](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781966831, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 14:47:11.207290 408 init.cc:377] Logged in from keytab as kudu/127.0.97.62@KRBTEST.COM (short username kudu)
I20260620 14:47:11.209049 408 webserver.cc:460] Webserver started at http://127.0.97.62:35993/ using document root <none> and password file <none>
I20260620 14:47:11.209800 408 fs_manager.cc:359] Metadata directory not provided
I20260620 14:47:11.209946 408 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 14:47:11.210264 408 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 14:47:11.213788 408 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskUci3DU/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781966829091300-388-0/minicluster-data/master-0/data/instance:
uuid: "5787471112774f63b76d05003e6ddc24"
format_stamp: "Formatted at 2026-06-20 14:47:11 on dist-test-slave-q9j6"
server_key: "fb3d263f9aeb7edeff421aabf7a87e82"
server_key_iv: "554f2a70d761449998f04f19fcfdd838"
server_key_version: "encryptionkey@0"
I20260620 14:47:11.214581 408 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskUci3DU/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781966829091300-388-0/minicluster-data/master-0/wal/instance:
uuid: "5787471112774f63b76d05003e6ddc24"
format_stamp: "Formatted at 2026-06-20 14:47:11 on dist-test-slave-q9j6"
server_key: "fb3d263f9aeb7edeff421aabf7a87e82"
server_key_iv: "554f2a70d761449998f04f19fcfdd838"
server_key_version: "encryptionkey@0"
I20260620 14:47:11.220104 408 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.007s sys 0.001s
I20260620 14:47:11.223739 423 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 14:47:11.225194 408 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260620 14:47:11.225430 408 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskUci3DU/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781966829091300-388-0/minicluster-data/master-0/data,/tmp/dist-test-taskUci3DU/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781966829091300-388-0/minicluster-data/master-0/wal
uuid: "5787471112774f63b76d05003e6ddc24"
format_stamp: "Formatted at 2026-06-20 14:47:11 on dist-test-slave-q9j6"
server_key: "fb3d263f9aeb7edeff421aabf7a87e82"
server_key_iv: "554f2a70d761449998f04f19fcfdd838"
server_key_version: "encryptionkey@0"
I20260620 14:47:11.225656 408 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskUci3DU/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781966829091300-388-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskUci3DU/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781966829091300-388-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskUci3DU/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781966829091300-388-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 14:47:11.247529 408 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 14:47:11.252238 408 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 14:47:11.252568 408 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 14:47:11.272843 408 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.62:35831
I20260620 14:47:11.272871 474 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.62:35831 every 8 connection(s)
I20260620 14:47:11.274343 408 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskUci3DU/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781966829091300-388-0/minicluster-data/master-0/data/info.pb
I20260620 14:47:11.278775 475 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 14:47:11.284925 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskUci3DU/build/asan/bin/kudu as pid 408
I20260620 14:47:11.285246 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskUci3DU/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781966829091300-388-0/minicluster-data/master-0/wal/instance
I20260620 14:47:11.287842 388 external_mini_cluster.cc:1428] Setting key d1170c15b0c154f4d5683081dd8254a8
I20260620 14:47:11.292115 475 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 5787471112774f63b76d05003e6ddc24: Bootstrap starting.
I20260620 14:47:11.296993 475 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 5787471112774f63b76d05003e6ddc24: Neither blocks nor log segments found. Creating new log.
I20260620 14:47:11.298437 475 log.cc:824] T 00000000000000000000000000000000 P 5787471112774f63b76d05003e6ddc24: Log is configured to *not* fsync() on all Append() calls
I20260620 14:47:11.302248 475 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 5787471112774f63b76d05003e6ddc24: No bootstrap required, opened a new log
I20260620 14:47:11.314524 475 raft_consensus.cc:348] T 00000000000000000000000000000000 P 5787471112774f63b76d05003e6ddc24 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5787471112774f63b76d05003e6ddc24" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 35831 } }
I20260620 14:47:11.314996 475 raft_consensus.cc:374] T 00000000000000000000000000000000 P 5787471112774f63b76d05003e6ddc24 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 14:47:11.315133 475 raft_consensus.cc:729] T 00000000000000000000000000000000 P 5787471112774f63b76d05003e6ddc24 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5787471112774f63b76d05003e6ddc24, State: Initialized, Role: FOLLOWER
I20260620 14:47:11.315860 475 consensus_queue.cc:260] T 00000000000000000000000000000000 P 5787471112774f63b76d05003e6ddc24 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5787471112774f63b76d05003e6ddc24" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 35831 } }
I20260620 14:47:11.316159 475 raft_consensus.cc:388] T 00000000000000000000000000000000 P 5787471112774f63b76d05003e6ddc24 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 14:47:11.316316 475 raft_consensus.cc:482] T 00000000000000000000000000000000 P 5787471112774f63b76d05003e6ddc24 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 14:47:11.316497 475 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 5787471112774f63b76d05003e6ddc24 [term 0 FOLLOWER]: Advancing to term 1
I20260620 14:47:11.319514 475 raft_consensus.cc:504] T 00000000000000000000000000000000 P 5787471112774f63b76d05003e6ddc24 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5787471112774f63b76d05003e6ddc24" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 35831 } }
I20260620 14:47:11.320218 475 leader_election.cc:302] T 00000000000000000000000000000000 P 5787471112774f63b76d05003e6ddc24 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 5787471112774f63b76d05003e6ddc24; no voters:
I20260620 14:47:11.321343 475 leader_election.cc:288] T 00000000000000000000000000000000 P 5787471112774f63b76d05003e6ddc24 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260620 14:47:11.321462 480 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 5787471112774f63b76d05003e6ddc24 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 14:47:11.322860 480 raft_consensus.cc:686] T 00000000000000000000000000000000 P 5787471112774f63b76d05003e6ddc24 [term 1 LEADER]: Becoming Leader. State: Replica: 5787471112774f63b76d05003e6ddc24, State: Running, Role: LEADER
I20260620 14:47:11.323390 480 consensus_queue.cc:237] T 00000000000000000000000000000000 P 5787471112774f63b76d05003e6ddc24 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5787471112774f63b76d05003e6ddc24" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 35831 } }
I20260620 14:47:11.324038 475 sys_catalog.cc:564] T 00000000000000000000000000000000 P 5787471112774f63b76d05003e6ddc24 [sys.catalog]: configured and running, proceeding with master startup.
Jun 20 14:47:11 dist-test-slave-q9j6 krb5kdc[395](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781966830, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260620 14:47:11.326251 482 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5787471112774f63b76d05003e6ddc24 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 5787471112774f63b76d05003e6ddc24. Latest consensus state: current_term: 1 leader_uuid: "5787471112774f63b76d05003e6ddc24" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5787471112774f63b76d05003e6ddc24" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 35831 } } }
I20260620 14:47:11.326646 482 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5787471112774f63b76d05003e6ddc24 [sys.catalog]: This master's current role is: LEADER
I20260620 14:47:11.327193 481 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5787471112774f63b76d05003e6ddc24 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "5787471112774f63b76d05003e6ddc24" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5787471112774f63b76d05003e6ddc24" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 35831 } } }
I20260620 14:47:11.328405 481 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5787471112774f63b76d05003e6ddc24 [sys.catalog]: This master's current role is: LEADER
I20260620 14:47:11.339637 485 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 14:47:11.347263 485 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 14:47:11.349318 477 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:47:11.292829 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44742 (local address 127.0.97.62:35831)
0620 14:47:11.293850 (+ 1021us) server_negotiation.cc:206] Beginning negotiation
0620 14:47:11.293870 (+ 20us) server_negotiation.cc:399] Waiting for connection header
0620 14:47:11.295295 (+ 1425us) server_negotiation.cc:407] Connection header received
0620 14:47:11.308802 (+ 13507us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:47:11.308831 (+ 29us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:47:11.309385 (+ 554us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:47:11.309803 (+ 418us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:47:11.314169 (+ 4366us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:47:11.315862 (+ 1693us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:47:11.318575 (+ 2713us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:47:11.319314 (+ 739us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:47:11.319478 (+ 164us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:47:11.335557 (+ 16079us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 14:47:11.335587 (+ 30us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 14:47:11.335657 (+ 70us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 14:47:11.335749 (+ 92us) server_negotiation.cc:922] calling sasl_server_start()
0620 14:47:11.340860 (+ 5111us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:47:11.341956 (+ 1096us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:47:11.341962 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:47:11.341969 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:47:11.342261 (+ 292us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:47:11.342984 (+ 723us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:47:11.342989 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:47:11.342991 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:47:11.343789 (+ 798us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 14:47:11.347002 (+ 3213us) server_negotiation.cc:1035] Waiting for connection context
0620 14:47:11.347314 (+ 312us) server_negotiation.cc:299] Negotiation successful
0620 14:47:11.347829 (+ 515us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":708,"thread_start_us":423,"threads_started":1}
I20260620 14:47:11.360431 485 catalog_manager.cc:1347] Generated new cluster ID: 35eee211c4b84deb89bb343b201e043b
I20260620 14:47:11.360560 485 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 14:47:11.393163 485 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 14:47:11.394596 485 catalog_manager.cc:1504] Loading token signing keys...
I20260620 14:47:11.404134 485 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 5787471112774f63b76d05003e6ddc24: Generated new TSK 0
I20260620 14:47:11.405055 485 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.0.97.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskUci3DU/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781966829091300-388-0/krb5kdc/kudu_127.0.97.1.keytab.
Entry for principal kudu/127.0.97.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskUci3DU/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781966829091300-388-0/krb5kdc/kudu_127.0.97.1.keytab.
I20260620 14:47:11.457015 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskUci3DU/build/asan/bin/kudu
/tmp/dist-test-taskUci3DU/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskUci3DU/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781966829091300-388-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskUci3DU/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781966829091300-388-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskUci3DU/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781966829091300-388-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskUci3DU/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781966829091300-388-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.1:0
--local_ip_for_outbound_sockets=127.0.97.1
--webserver_interface=127.0.97.1
--webserver_port=0
--tserver_master_addrs=127.0.97.62:35831
--keytab_file=/tmp/dist-test-taskUci3DU/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781966829091300-388-0/krb5kdc/kudu_127.0.97.1.keytab
--principal=kudu/127.0.97.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:43677
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-taskUci3DU/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781966829091300-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskUci3DU/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781966829091300-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskUci3DU/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781966829091300-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 14:47:11.803479 502 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 14:47:11.803917 502 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 14:47:11.804111 502 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 14:47:11.812070 502 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 14:47:11.812192 502 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 14:47:11.812397 502 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.1
I20260620 14:47:11.822461 502 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:43677
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskUci3DU/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781966829091300-388-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskUci3DU/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781966829091300-388-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskUci3DU/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781966829091300-388-0/krb5kdc/kudu_127.0.97.1.keytab
--principal=kudu/127.0.97.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskUci3DU/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781966829091300-388-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.97.1
--webserver_port=0
--tserver_master_addrs=127.0.97.62:35831
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.1
--log_dir=/tmp/dist-test-taskUci3DU/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781966829091300-388-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 13:41:20 UTC on ccycloud.quasar-vdagoi.root.comops.site
build id 4457462
ASAN enabled
I20260620 14:47:11.823906 502 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 14:47:11.825397 502 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 14:47:11.839555 510 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 14:47:11.839623 507 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 14:47:11.839512 508 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 14:47:11.840791 502 server_base.cc:1034] running on GCE node
I20260620 14:47:11.841512 502 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 14:47:11.842657 502 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 14:47:11.843885 502 hybrid_clock.cc:648] HybridClock initialized: now 1781966831843849 us; error 64 us; skew 500 ppm
I20260620 14:47:11.844236 502 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 14:47:11 dist-test-slave-q9j6 krb5kdc[395](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781966831, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 14:47:11.847384 502 init.cc:377] Logged in from keytab as kudu/127.0.97.1@KRBTEST.COM (short username kudu)
I20260620 14:47:11.848897 502 webserver.cc:460] Webserver started at http://127.0.97.1:36981/ using document root <none> and password file <none>
I20260620 14:47:11.849615 502 fs_manager.cc:359] Metadata directory not provided
I20260620 14:47:11.849753 502 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 14:47:11.850060 502 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 14:47:11.853080 502 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskUci3DU/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781966829091300-388-0/minicluster-data/ts-0/data/instance:
uuid: "22f541443c4e4173af2a9e4fd7df673c"
format_stamp: "Formatted at 2026-06-20 14:47:11 on dist-test-slave-q9j6"
server_key: "bd0f113175e3f374fcf678bbd8e3380e"
server_key_iv: "3efe3ef96f336db90e8ea329c7067a26"
server_key_version: "encryptionkey@0"
I20260620 14:47:11.853899 502 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskUci3DU/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781966829091300-388-0/minicluster-data/ts-0/wal/instance:
uuid: "22f541443c4e4173af2a9e4fd7df673c"
format_stamp: "Formatted at 2026-06-20 14:47:11 on dist-test-slave-q9j6"
server_key: "bd0f113175e3f374fcf678bbd8e3380e"
server_key_iv: "3efe3ef96f336db90e8ea329c7067a26"
server_key_version: "encryptionkey@0"
I20260620 14:47:11.859266 502 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.001s
I20260620 14:47:11.863454 517 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 14:47:11.865067 502 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.000s sys 0.003s
I20260620 14:47:11.865293 502 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskUci3DU/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781966829091300-388-0/minicluster-data/ts-0/data,/tmp/dist-test-taskUci3DU/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781966829091300-388-0/minicluster-data/ts-0/wal
uuid: "22f541443c4e4173af2a9e4fd7df673c"
format_stamp: "Formatted at 2026-06-20 14:47:11 on dist-test-slave-q9j6"
server_key: "bd0f113175e3f374fcf678bbd8e3380e"
server_key_iv: "3efe3ef96f336db90e8ea329c7067a26"
server_key_version: "encryptionkey@0"
I20260620 14:47:11.865579 502 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskUci3DU/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781966829091300-388-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskUci3DU/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781966829091300-388-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskUci3DU/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781966829091300-388-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 14:47:11.882756 502 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 14:47:11.887495 502 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 14:47:11.887868 502 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 14:47:11.888883 502 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 14:47:11.890234 502 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 14:47:11.890306 502 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 14:47:11.890466 502 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 14:47:11.890962 502 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 14:47:11.927016 502 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.1:37557
I20260620 14:47:11.927037 629 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.1:37557 every 8 connection(s)
I20260620 14:47:11.928500 502 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskUci3DU/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781966829091300-388-0/minicluster-data/ts-0/data/info.pb
I20260620 14:47:11.936554 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskUci3DU/build/asan/bin/kudu as pid 502
I20260620 14:47:11.936792 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskUci3DU/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781966829091300-388-0/minicluster-data/ts-0/wal/instance
I20260620 14:47:11.937435 388 external_mini_cluster.cc:1428] Setting key 97253b1b5fc9d95ed6dc5291f2c91224
Jun 20 14:47:11 dist-test-slave-q9j6 krb5kdc[395](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781966831, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.1@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260620 14:47:11.951722 632 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:47:11.931294 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.1:59431 (local address 127.0.97.62:35831)
0620 14:47:11.931765 (+ 471us) server_negotiation.cc:206] Beginning negotiation
0620 14:47:11.931770 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 14:47:11.932814 (+ 1044us) server_negotiation.cc:407] Connection header received
0620 14:47:11.934694 (+ 1880us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:47:11.934700 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:47:11.934795 (+ 95us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:47:11.934919 (+ 124us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:47:11.936572 (+ 1653us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:47:11.937402 (+ 830us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:47:11.939599 (+ 2197us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:47:11.940266 (+ 667us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:47:11.940430 (+ 164us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:47:11.944854 (+ 4424us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 14:47:11.944859 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 14:47:11.944863 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 14:47:11.944897 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0620 14:47:11.948040 (+ 3143us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:47:11.949105 (+ 1065us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:47:11.949109 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:47:11.949112 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:47:11.949232 (+ 120us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:47:11.949757 (+ 525us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:47:11.949760 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:47:11.949763 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:47:11.950018 (+ 255us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 14:47:11.950127 (+ 109us) server_negotiation.cc:1035] Waiting for connection context
0620 14:47:11.951108 (+ 981us) server_negotiation.cc:299] Negotiation successful
0620 14:47:11.951382 (+ 274us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":372,"thread_start_us":242,"threads_started":1}
I20260620 14:47:11.953331 633 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:47:11.931896 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:35831 (local address 127.0.97.1:59431)
0620 14:47:11.932628 (+ 732us) negotiation.cc:107] Waiting for socket to connect
0620 14:47:11.932675 (+ 47us) client_negotiation.cc:174] Beginning negotiation
0620 14:47:11.934395 (+ 1720us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 14:47:11.935174 (+ 779us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 14:47:11.935182 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 14:47:11.935805 (+ 623us) client_negotiation.cc:189] Negotiated authn=SASL
0620 14:47:11.936393 (+ 588us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:47:11.936410 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:47:11.937577 (+ 1167us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:47:11.937587 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:47:11.939346 (+ 1759us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:47:11.939361 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:47:11.940505 (+ 1144us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:47:11.940510 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:47:11.940749 (+ 239us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:47:11.941898 (+ 1149us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 14:47:11.941924 (+ 26us) client_negotiation.cc:656] calling sasl_client_start()
0620 14:47:11.944548 (+ 2624us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 14:47:11.948219 (+ 3671us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 14:47:11.948233 (+ 14us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 14:47:11.948258 (+ 25us) client_negotiation.cc:760] calling sasl_client_step()
0620 14:47:11.948962 (+ 704us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 14:47:11.949344 (+ 382us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 14:47:11.949351 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 14:47:11.949358 (+ 7us) client_negotiation.cc:760] calling sasl_client_step()
0620 14:47:11.949637 (+ 279us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 14:47:11.950152 (+ 515us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 14:47:11.950163 (+ 11us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 14:47:11.950676 (+ 513us) client_negotiation.cc:769] Sending connection context
0620 14:47:11.951010 (+ 334us) client_negotiation.cc:240] Negotiation successful
0620 14:47:11.951685 (+ 675us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":485,"thread_start_us":287,"threads_started":1}
I20260620 14:47:11.955176 630 heartbeater.cc:347] Connected to a master server at 127.0.97.62:35831
I20260620 14:47:11.955518 630 heartbeater.cc:464] Registering TS with master...
WARNING: no policy specified for kudu/127.0.97.2@KRBTEST.COM; defaulting to no policy
I20260620 14:47:11.956410 630 heartbeater.cc:511] Master 127.0.97.62:35831 requested a full tablet report, sending...
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.2@KRBTEST.COM" created.
I20260620 14:47:11.958773 440 ts_manager.cc:194] Registered new tserver with Master: 22f541443c4e4173af2a9e4fd7df673c (127.0.97.1:37557)
I20260620 14:47:11.960304 440 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.97.1@KRBTEST.COM'} at 127.0.97.1:59431
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskUci3DU/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781966829091300-388-0/krb5kdc/kudu_127.0.97.2.keytab.
Entry for principal kudu/127.0.97.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskUci3DU/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781966829091300-388-0/krb5kdc/kudu_127.0.97.2.keytab.
I20260620 14:47:11.978935 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskUci3DU/build/asan/bin/kudu
/tmp/dist-test-taskUci3DU/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskUci3DU/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781966829091300-388-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskUci3DU/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781966829091300-388-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskUci3DU/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781966829091300-388-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskUci3DU/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781966829091300-388-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.2:0
--local_ip_for_outbound_sockets=127.0.97.2
--webserver_interface=127.0.97.2
--webserver_port=0
--tserver_master_addrs=127.0.97.62:35831
--keytab_file=/tmp/dist-test-taskUci3DU/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781966829091300-388-0/krb5kdc/kudu_127.0.97.2.keytab
--principal=kudu/127.0.97.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:43677
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-taskUci3DU/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781966829091300-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskUci3DU/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781966829091300-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskUci3DU/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781966829091300-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 14:47:12.329584 636 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 14:47:12.329873 636 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 14:47:12.330044 636 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 14:47:12.337322 636 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 14:47:12.337457 636 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 14:47:12.337652 636 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.2
I20260620 14:47:12.347846 636 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:43677
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskUci3DU/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781966829091300-388-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskUci3DU/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781966829091300-388-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskUci3DU/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781966829091300-388-0/krb5kdc/kudu_127.0.97.2.keytab
--principal=kudu/127.0.97.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskUci3DU/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781966829091300-388-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.97.2
--webserver_port=0
--tserver_master_addrs=127.0.97.62:35831
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.2
--log_dir=/tmp/dist-test-taskUci3DU/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781966829091300-388-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 13:41:20 UTC on ccycloud.quasar-vdagoi.root.comops.site
build id 4457462
ASAN enabled
I20260620 14:47:12.349612 636 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 14:47:12.351006 636 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 14:47:12.364740 641 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 14:47:12.364781 642 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 14:47:12.364790 644 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 14:47:12.365427 636 server_base.cc:1034] running on GCE node
I20260620 14:47:12.366060 636 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 14:47:12.367026 636 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 14:47:12.368252 636 hybrid_clock.cc:648] HybridClock initialized: now 1781966832368206 us; error 40 us; skew 500 ppm
I20260620 14:47:12.368574 636 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 14:47:12 dist-test-slave-q9j6 krb5kdc[395](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781966832, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 14:47:12.372239 636 init.cc:377] Logged in from keytab as kudu/127.0.97.2@KRBTEST.COM (short username kudu)
I20260620 14:47:12.373698 636 webserver.cc:460] Webserver started at http://127.0.97.2:45475/ using document root <none> and password file <none>
I20260620 14:47:12.374428 636 fs_manager.cc:359] Metadata directory not provided
I20260620 14:47:12.374568 636 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 14:47:12.374899 636 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 14:47:12.377934 636 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskUci3DU/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781966829091300-388-0/minicluster-data/ts-1/data/instance:
uuid: "f676bd4820ef4035866761c3a7752888"
format_stamp: "Formatted at 2026-06-20 14:47:12 on dist-test-slave-q9j6"
server_key: "301df3a4e7c9af883886c29e86ecbd0f"
server_key_iv: "6adf95f0b91ab71e93bbc9f48e4d1333"
server_key_version: "encryptionkey@0"
I20260620 14:47:12.378775 636 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskUci3DU/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781966829091300-388-0/minicluster-data/ts-1/wal/instance:
uuid: "f676bd4820ef4035866761c3a7752888"
format_stamp: "Formatted at 2026-06-20 14:47:12 on dist-test-slave-q9j6"
server_key: "301df3a4e7c9af883886c29e86ecbd0f"
server_key_iv: "6adf95f0b91ab71e93bbc9f48e4d1333"
server_key_version: "encryptionkey@0"
I20260620 14:47:12.384115 636 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.007s sys 0.000s
I20260620 14:47:12.387560 651 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 14:47:12.388981 636 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260620 14:47:12.389181 636 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskUci3DU/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781966829091300-388-0/minicluster-data/ts-1/data,/tmp/dist-test-taskUci3DU/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781966829091300-388-0/minicluster-data/ts-1/wal
uuid: "f676bd4820ef4035866761c3a7752888"
format_stamp: "Formatted at 2026-06-20 14:47:12 on dist-test-slave-q9j6"
server_key: "301df3a4e7c9af883886c29e86ecbd0f"
server_key_iv: "6adf95f0b91ab71e93bbc9f48e4d1333"
server_key_version: "encryptionkey@0"
I20260620 14:47:12.389393 636 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskUci3DU/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781966829091300-388-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskUci3DU/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781966829091300-388-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskUci3DU/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781966829091300-388-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 14:47:12.406307 636 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 14:47:12.410549 636 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 14:47:12.410861 636 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 14:47:12.411876 636 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 14:47:12.413180 636 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 14:47:12.413252 636 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 14:47:12.413422 636 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 14:47:12.413481 636 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 14:47:12.448544 636 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.2:34851
I20260620 14:47:12.448580 763 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.2:34851 every 8 connection(s)
I20260620 14:47:12.449874 636 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskUci3DU/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781966829091300-388-0/minicluster-data/ts-1/data/info.pb
I20260620 14:47:12.456625 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskUci3DU/build/asan/bin/kudu as pid 636
I20260620 14:47:12.456851 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskUci3DU/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781966829091300-388-0/minicluster-data/ts-1/wal/instance
I20260620 14:47:12.457373 388 external_mini_cluster.cc:1428] Setting key 1a37d98ecde385a212ace8b4acc69725
Jun 20 14:47:12 dist-test-slave-q9j6 krb5kdc[395](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781966832, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.2@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260620 14:47:12.473932 766 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:47:12.452865 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.2:56697 (local address 127.0.97.62:35831)
0620 14:47:12.453409 (+ 544us) server_negotiation.cc:206] Beginning negotiation
0620 14:47:12.453419 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0620 14:47:12.454073 (+ 654us) server_negotiation.cc:407] Connection header received
0620 14:47:12.455985 (+ 1912us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:47:12.455990 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:47:12.456058 (+ 68us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:47:12.456165 (+ 107us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:47:12.458302 (+ 2137us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:47:12.459134 (+ 832us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:47:12.461788 (+ 2654us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:47:12.462244 (+ 456us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:47:12.462348 (+ 104us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:47:12.466752 (+ 4404us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 14:47:12.466762 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 14:47:12.466770 (+ 8us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 14:47:12.466816 (+ 46us) server_negotiation.cc:922] calling sasl_server_start()
0620 14:47:12.470526 (+ 3710us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:47:12.471405 (+ 879us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:47:12.471413 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:47:12.471421 (+ 8us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:47:12.471552 (+ 131us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:47:12.472138 (+ 586us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:47:12.472141 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:47:12.472143 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:47:12.472408 (+ 265us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 14:47:12.472525 (+ 117us) server_negotiation.cc:1035] Waiting for connection context
0620 14:47:12.473511 (+ 986us) server_negotiation.cc:299] Negotiation successful
0620 14:47:12.473674 (+ 163us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":372,"thread_start_us":261,"threads_started":1}
I20260620 14:47:12.476058 767 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:47:12.452968 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:35831 (local address 127.0.97.2:56697)
0620 14:47:12.453837 (+ 869us) negotiation.cc:107] Waiting for socket to connect
0620 14:47:12.453896 (+ 59us) client_negotiation.cc:174] Beginning negotiation
0620 14:47:12.455720 (+ 1824us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 14:47:12.456510 (+ 790us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 14:47:12.456522 (+ 12us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 14:47:12.457308 (+ 786us) client_negotiation.cc:189] Negotiated authn=SASL
0620 14:47:12.458112 (+ 804us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:47:12.458132 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:47:12.459308 (+ 1176us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:47:12.459312 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:47:12.461600 (+ 2288us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:47:12.461622 (+ 22us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:47:12.462519 (+ 897us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:47:12.462528 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:47:12.462877 (+ 349us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:47:12.463693 (+ 816us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 14:47:12.463729 (+ 36us) client_negotiation.cc:656] calling sasl_client_start()
0620 14:47:12.466475 (+ 2746us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 14:47:12.470688 (+ 4213us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 14:47:12.470695 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 14:47:12.470711 (+ 16us) client_negotiation.cc:760] calling sasl_client_step()
0620 14:47:12.471189 (+ 478us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 14:47:12.471720 (+ 531us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 14:47:12.471725 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 14:47:12.471730 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0620 14:47:12.472011 (+ 281us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 14:47:12.472541 (+ 530us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 14:47:12.472553 (+ 12us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 14:47:12.473107 (+ 554us) client_negotiation.cc:769] Sending connection context
0620 14:47:12.473446 (+ 339us) client_negotiation.cc:240] Negotiation successful
0620 14:47:12.474064 (+ 618us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":547,"thread_start_us":342,"threads_started":1}
WARNING: no policy specified for kudu/127.0.97.3@KRBTEST.COM; defaulting to no policy
I20260620 14:47:12.477746 764 heartbeater.cc:347] Connected to a master server at 127.0.97.62:35831
I20260620 14:47:12.478061 764 heartbeater.cc:464] Registering TS with master...
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.3@KRBTEST.COM" created.
I20260620 14:47:12.478932 764 heartbeater.cc:511] Master 127.0.97.62:35831 requested a full tablet report, sending...
I20260620 14:47:12.480724 440 ts_manager.cc:194] Registered new tserver with Master: f676bd4820ef4035866761c3a7752888 (127.0.97.2:34851)
I20260620 14:47:12.481822 440 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.97.2@KRBTEST.COM'} at 127.0.97.2:56697
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskUci3DU/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781966829091300-388-0/krb5kdc/kudu_127.0.97.3.keytab.
Entry for principal kudu/127.0.97.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskUci3DU/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781966829091300-388-0/krb5kdc/kudu_127.0.97.3.keytab.
I20260620 14:47:12.500689 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskUci3DU/build/asan/bin/kudu
/tmp/dist-test-taskUci3DU/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskUci3DU/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781966829091300-388-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskUci3DU/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781966829091300-388-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskUci3DU/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781966829091300-388-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskUci3DU/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781966829091300-388-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.3:0
--local_ip_for_outbound_sockets=127.0.97.3
--webserver_interface=127.0.97.3
--webserver_port=0
--tserver_master_addrs=127.0.97.62:35831
--keytab_file=/tmp/dist-test-taskUci3DU/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781966829091300-388-0/krb5kdc/kudu_127.0.97.3.keytab
--principal=kudu/127.0.97.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:43677
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-taskUci3DU/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781966829091300-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskUci3DU/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781966829091300-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskUci3DU/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781966829091300-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 14:47:12.848304 770 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 14:47:12.848577 770 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 14:47:12.848773 770 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 14:47:12.856117 770 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 14:47:12.856240 770 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 14:47:12.856419 770 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.3
I20260620 14:47:12.866792 770 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:43677
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskUci3DU/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781966829091300-388-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskUci3DU/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781966829091300-388-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskUci3DU/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781966829091300-388-0/krb5kdc/kudu_127.0.97.3.keytab
--principal=kudu/127.0.97.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskUci3DU/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781966829091300-388-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.97.3
--webserver_port=0
--tserver_master_addrs=127.0.97.62:35831
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.3
--log_dir=/tmp/dist-test-taskUci3DU/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781966829091300-388-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 13:41:20 UTC on ccycloud.quasar-vdagoi.root.comops.site
build id 4457462
ASAN enabled
I20260620 14:47:12.868260 770 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 14:47:12.870203 770 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 14:47:12.884285 776 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 14:47:12.884224 778 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 14:47:12.884240 775 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 14:47:12.885087 770 server_base.cc:1034] running on GCE node
I20260620 14:47:12.885787 770 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 14:47:12.886838 770 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 14:47:12.888474 770 hybrid_clock.cc:648] HybridClock initialized: now 1781966832887947 us; error 122 us; skew 500 ppm
I20260620 14:47:12.888832 770 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 14:47:12 dist-test-slave-q9j6 krb5kdc[395](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781966832, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 14:47:12.891976 770 init.cc:377] Logged in from keytab as kudu/127.0.97.3@KRBTEST.COM (short username kudu)
I20260620 14:47:12.893622 770 webserver.cc:460] Webserver started at http://127.0.97.3:41943/ using document root <none> and password file <none>
I20260620 14:47:12.894464 770 fs_manager.cc:359] Metadata directory not provided
I20260620 14:47:12.894604 770 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 14:47:12.894909 770 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 14:47:12.898038 770 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskUci3DU/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781966829091300-388-0/minicluster-data/ts-2/data/instance:
uuid: "80ccdc417701475aa339608c894b1b06"
format_stamp: "Formatted at 2026-06-20 14:47:12 on dist-test-slave-q9j6"
server_key: "62ecbac5cbcbb64dd884ffd2f96994c0"
server_key_iv: "21efb36f67c73b686909a039364dd9ff"
server_key_version: "encryptionkey@0"
I20260620 14:47:12.898864 770 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskUci3DU/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781966829091300-388-0/minicluster-data/ts-2/wal/instance:
uuid: "80ccdc417701475aa339608c894b1b06"
format_stamp: "Formatted at 2026-06-20 14:47:12 on dist-test-slave-q9j6"
server_key: "62ecbac5cbcbb64dd884ffd2f96994c0"
server_key_iv: "21efb36f67c73b686909a039364dd9ff"
server_key_version: "encryptionkey@0"
I20260620 14:47:12.904253 770 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.007s sys 0.001s
I20260620 14:47:12.907998 785 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 14:47:12.909443 770 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260620 14:47:12.909639 770 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskUci3DU/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781966829091300-388-0/minicluster-data/ts-2/data,/tmp/dist-test-taskUci3DU/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781966829091300-388-0/minicluster-data/ts-2/wal
uuid: "80ccdc417701475aa339608c894b1b06"
format_stamp: "Formatted at 2026-06-20 14:47:12 on dist-test-slave-q9j6"
server_key: "62ecbac5cbcbb64dd884ffd2f96994c0"
server_key_iv: "21efb36f67c73b686909a039364dd9ff"
server_key_version: "encryptionkey@0"
I20260620 14:47:12.909860 770 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskUci3DU/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781966829091300-388-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskUci3DU/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781966829091300-388-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskUci3DU/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781966829091300-388-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 14:47:12.926744 770 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 14:47:12.931627 770 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 14:47:12.931924 770 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 14:47:12.932909 770 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 14:47:12.934290 770 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 14:47:12.934401 770 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 14:47:12.934553 770 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 14:47:12.934629 770 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 14:47:12.963572 630 heartbeater.cc:503] Master 127.0.97.62:35831 was elected leader, sending a full tablet report...
I20260620 14:47:12.971652 770 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.3:35147
I20260620 14:47:12.971763 897 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.3:35147 every 8 connection(s)
I20260620 14:47:12.972992 770 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskUci3DU/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781966829091300-388-0/minicluster-data/ts-2/data/info.pb
I20260620 14:47:12.978786 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskUci3DU/build/asan/bin/kudu as pid 770
I20260620 14:47:12.978979 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskUci3DU/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781966829091300-388-0/minicluster-data/ts-2/wal/instance
I20260620 14:47:12.979400 388 external_mini_cluster.cc:1428] Setting key 48c690efe1e19c67f2aed5f8d343beea
Jun 20 14:47:12 dist-test-slave-q9j6 krb5kdc[395](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781966832, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.3@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260620 14:47:12.993937 900 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:47:12.975708 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.3:38597 (local address 127.0.97.62:35831)
0620 14:47:12.976162 (+ 454us) server_negotiation.cc:206] Beginning negotiation
0620 14:47:12.976166 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 14:47:12.977302 (+ 1136us) server_negotiation.cc:407] Connection header received
0620 14:47:12.979236 (+ 1934us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:47:12.979241 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:47:12.979339 (+ 98us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:47:12.979468 (+ 129us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:47:12.981215 (+ 1747us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:47:12.982028 (+ 813us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:47:12.983648 (+ 1620us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:47:12.984048 (+ 400us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:47:12.984192 (+ 144us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:47:12.987728 (+ 3536us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 14:47:12.987733 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 14:47:12.987736 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 14:47:12.987769 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0620 14:47:12.990818 (+ 3049us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:47:12.991674 (+ 856us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:47:12.991679 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:47:12.991681 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:47:12.991800 (+ 119us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:47:12.992179 (+ 379us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:47:12.992182 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:47:12.992184 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:47:12.992430 (+ 246us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 14:47:12.992535 (+ 105us) server_negotiation.cc:1035] Waiting for connection context
0620 14:47:12.993278 (+ 743us) server_negotiation.cc:299] Negotiation successful
0620 14:47:12.993558 (+ 280us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":354,"thread_start_us":229,"threads_started":1}
I20260620 14:47:12.995034 901 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:47:12.976252 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:35831 (local address 127.0.97.3:38597)
0620 14:47:12.977120 (+ 868us) negotiation.cc:107] Waiting for socket to connect
0620 14:47:12.977168 (+ 48us) client_negotiation.cc:174] Beginning negotiation
0620 14:47:12.978887 (+ 1719us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 14:47:12.979760 (+ 873us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 14:47:12.979770 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 14:47:12.980319 (+ 549us) client_negotiation.cc:189] Negotiated authn=SASL
0620 14:47:12.981042 (+ 723us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:47:12.981062 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:47:12.982205 (+ 1143us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:47:12.982209 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:47:12.983462 (+ 1253us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:47:12.983474 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:47:12.984286 (+ 812us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:47:12.984294 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:47:12.984543 (+ 249us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:47:12.985227 (+ 684us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 14:47:12.985250 (+ 23us) client_negotiation.cc:656] calling sasl_client_start()
0620 14:47:12.987459 (+ 2209us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 14:47:12.991027 (+ 3568us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 14:47:12.991034 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 14:47:12.991048 (+ 14us) client_negotiation.cc:760] calling sasl_client_step()
0620 14:47:12.991496 (+ 448us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 14:47:12.991899 (+ 403us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 14:47:12.991902 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 14:47:12.991904 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 14:47:12.992086 (+ 182us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 14:47:12.992548 (+ 462us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 14:47:12.992554 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 14:47:12.992904 (+ 350us) client_negotiation.cc:769] Sending connection context
0620 14:47:12.993137 (+ 233us) client_negotiation.cc:240] Negotiation successful
0620 14:47:12.993704 (+ 567us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":607,"thread_start_us":377,"threads_started":1}
I20260620 14:47:12.996428 898 heartbeater.cc:347] Connected to a master server at 127.0.97.62:35831
I20260620 14:47:12.996697 898 heartbeater.cc:464] Registering TS with master...
I20260620 14:47:12.997421 898 heartbeater.cc:511] Master 127.0.97.62:35831 requested a full tablet report, sending...
I20260620 14:47:12.998826 440 ts_manager.cc:194] Registered new tserver with Master: 80ccdc417701475aa339608c894b1b06 (127.0.97.3:35147)
I20260620 14:47:12.999732 440 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.97.3@KRBTEST.COM'} at 127.0.97.3:38597
I20260620 14:47:13.008347 388 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
Jun 20 14:47:13 dist-test-slave-q9j6 krb5kdc[395](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781966833, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-user@KRBTEST.COM:
Jun 20 14:47:13 dist-test-slave-q9j6 krb5kdc[395](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781966833, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.0.97.1@KRBTEST.COM
I20260620 14:47:13.054145 906 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:47:13.038296 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46420 (local address 127.0.97.1:37557)
0620 14:47:13.038944 (+ 648us) server_negotiation.cc:206] Beginning negotiation
0620 14:47:13.038952 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0620 14:47:13.038980 (+ 28us) server_negotiation.cc:407] Connection header received
0620 14:47:13.039110 (+ 130us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:47:13.039117 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:47:13.039356 (+ 239us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:47:13.039490 (+ 134us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:47:13.041035 (+ 1545us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:47:13.042099 (+ 1064us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:47:13.043364 (+ 1265us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:47:13.044050 (+ 686us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:47:13.044171 (+ 121us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:47:13.048846 (+ 4675us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 14:47:13.048860 (+ 14us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 14:47:13.048876 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 14:47:13.048916 (+ 40us) server_negotiation.cc:922] calling sasl_server_start()
0620 14:47:13.051700 (+ 2784us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:47:13.052338 (+ 638us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:47:13.052345 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:47:13.052347 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:47:13.052462 (+ 115us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:47:13.052769 (+ 307us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:47:13.052772 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:47:13.052774 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:47:13.053096 (+ 322us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 14:47:13.053303 (+ 207us) server_negotiation.cc:1035] Waiting for connection context
0620 14:47:13.053525 (+ 222us) server_negotiation.cc:299] Negotiation successful
0620 14:47:13.053734 (+ 209us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":502,"thread_start_us":332,"threads_started":1}
W20260620 14:47:13.055474 545 server_base.cc:1116] Unauthorized access attempt to method kudu.tserver.TabletServerService.ListTablets from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:46420
Jun 20 14:47:13 dist-test-slave-q9j6 krb5kdc[395](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781966833, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
Jun 20 14:47:13 dist-test-slave-q9j6 krb5kdc[395](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781966833, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.1@KRBTEST.COM
I20260620 14:47:13.099952 906 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:47:13.085597 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46430 (local address 127.0.97.1:37557)
0620 14:47:13.085844 (+ 247us) server_negotiation.cc:206] Beginning negotiation
0620 14:47:13.085851 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0620 14:47:13.086147 (+ 296us) server_negotiation.cc:407] Connection header received
0620 14:47:13.086360 (+ 213us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:47:13.086370 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:47:13.086433 (+ 63us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:47:13.086539 (+ 106us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:47:13.087937 (+ 1398us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:47:13.088622 (+ 685us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:47:13.089829 (+ 1207us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:47:13.090253 (+ 424us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:47:13.090388 (+ 135us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:47:13.094837 (+ 4449us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 14:47:13.094843 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 14:47:13.094846 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 14:47:13.094882 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0620 14:47:13.097402 (+ 2520us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:47:13.098056 (+ 654us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:47:13.098062 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:47:13.098066 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:47:13.098191 (+ 125us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:47:13.098566 (+ 375us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:47:13.098571 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:47:13.098573 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:47:13.098818 (+ 245us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 14:47:13.099051 (+ 233us) server_negotiation.cc:1035] Waiting for connection context
0620 14:47:13.099369 (+ 318us) server_negotiation.cc:299] Negotiation successful
0620 14:47:13.099585 (+ 216us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":86}
I20260620 14:47:13.101498 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskUci3DU/build/asan/bin/kudu pid 502
I20260620 14:47:13.114738 906 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:47:13.102438 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46432 (local address 127.0.97.1:37557)
0620 14:47:13.102627 (+ 189us) server_negotiation.cc:206] Beginning negotiation
0620 14:47:13.102635 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0620 14:47:13.102956 (+ 321us) server_negotiation.cc:407] Connection header received
0620 14:47:13.103147 (+ 191us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:47:13.103159 (+ 12us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:47:13.103216 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:47:13.103313 (+ 97us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:47:13.104569 (+ 1256us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:47:13.105268 (+ 699us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:47:13.106505 (+ 1237us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:47:13.106957 (+ 452us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:47:13.107091 (+ 134us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:47:13.109500 (+ 2409us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 14:47:13.109506 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 14:47:13.109509 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 14:47:13.109543 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0620 14:47:13.112027 (+ 2484us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:47:13.112739 (+ 712us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:47:13.112746 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:47:13.112750 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:47:13.112891 (+ 141us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:47:13.113277 (+ 386us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:47:13.113283 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:47:13.113286 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:47:13.113545 (+ 259us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 14:47:13.113812 (+ 267us) server_negotiation.cc:1035] Waiting for connection context
0620 14:47:13.114123 (+ 311us) server_negotiation.cc:299] Negotiation successful
0620 14:47:13.114343 (+ 220us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":57}
I20260620 14:47:13.115585 625 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 14:47:13.198827 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskUci3DU/build/asan/bin/kudu with pid 502
I20260620 14:47:13.215008 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskUci3DU/build/asan/bin/kudu pid 636
Jun 20 14:47:13 dist-test-slave-q9j6 krb5kdc[395](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781966833, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.2@KRBTEST.COM
I20260620 14:47:13.231001 913 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:47:13.215951 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54286 (local address 127.0.97.2:34851)
0620 14:47:13.216452 (+ 501us) server_negotiation.cc:206] Beginning negotiation
0620 14:47:13.216459 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0620 14:47:13.216483 (+ 24us) server_negotiation.cc:407] Connection header received
0620 14:47:13.216545 (+ 62us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:47:13.216551 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:47:13.216757 (+ 206us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:47:13.216939 (+ 182us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:47:13.218102 (+ 1163us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:47:13.219149 (+ 1047us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:47:13.220579 (+ 1430us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:47:13.221147 (+ 568us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:47:13.221298 (+ 151us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:47:13.225518 (+ 4220us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 14:47:13.225531 (+ 13us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 14:47:13.225546 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 14:47:13.225585 (+ 39us) server_negotiation.cc:922] calling sasl_server_start()
0620 14:47:13.228323 (+ 2738us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:47:13.229000 (+ 677us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:47:13.229007 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:47:13.229009 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:47:13.229129 (+ 120us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:47:13.229589 (+ 460us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:47:13.229592 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:47:13.229594 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:47:13.229931 (+ 337us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 14:47:13.230148 (+ 217us) server_negotiation.cc:1035] Waiting for connection context
0620 14:47:13.230465 (+ 317us) server_negotiation.cc:299] Negotiation successful
0620 14:47:13.230721 (+ 256us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":383,"thread_start_us":273,"threads_started":1}
I20260620 14:47:13.232391 759 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 14:47:13.314999 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskUci3DU/build/asan/bin/kudu with pid 636
I20260620 14:47:13.338053 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskUci3DU/build/asan/bin/kudu pid 770
Jun 20 14:47:13 dist-test-slave-q9j6 krb5kdc[395](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781966833, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.3@KRBTEST.COM
I20260620 14:47:13.354395 915 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:47:13.338995 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33310 (local address 127.0.97.3:35147)
0620 14:47:13.339698 (+ 703us) server_negotiation.cc:206] Beginning negotiation
0620 14:47:13.339706 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0620 14:47:13.339737 (+ 31us) server_negotiation.cc:407] Connection header received
0620 14:47:13.339801 (+ 64us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:47:13.339808 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:47:13.340020 (+ 212us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:47:13.340220 (+ 200us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:47:13.341549 (+ 1329us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:47:13.342580 (+ 1031us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:47:13.343958 (+ 1378us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:47:13.344523 (+ 565us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:47:13.344667 (+ 144us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:47:13.348990 (+ 4323us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 14:47:13.349003 (+ 13us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 14:47:13.349020 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 14:47:13.349059 (+ 39us) server_negotiation.cc:922] calling sasl_server_start()
0620 14:47:13.351836 (+ 2777us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:47:13.352528 (+ 692us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:47:13.352536 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:47:13.352538 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:47:13.352655 (+ 117us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:47:13.352973 (+ 318us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:47:13.352976 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:47:13.352978 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:47:13.353312 (+ 334us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 14:47:13.353528 (+ 216us) server_negotiation.cc:1035] Waiting for connection context
0620 14:47:13.353788 (+ 260us) server_negotiation.cc:299] Negotiation successful
0620 14:47:13.353982 (+ 194us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":499,"thread_start_us":300,"threads_started":1}
I20260620 14:47:13.355880 893 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 14:47:13.438418 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskUci3DU/build/asan/bin/kudu with pid 770
I20260620 14:47:13.454957 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskUci3DU/build/asan/bin/kudu pid 408
I20260620 14:47:13.456262 470 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 14:47:13.558018 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskUci3DU/build/asan/bin/kudu with pid 408
2026-06-20T14:47:13Z chronyd exiting
[ OK ] SecurityITest.TestAuthorizationOnListTablets (4426 ms)
[ RUN ] SecurityITest.TestAuthorizationOnChecksum
Loading random data
Initializing database '/tmp/dist-test-taskUci3DU/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781966829091300-388-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 20 14:47:13 dist-test-slave-q9j6 krb5kdc[920](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 20 14:47:13 dist-test-slave-q9j6 krb5kdc[920](info): set up 2 sockets
Jun 20 14:47:13 dist-test-slave-q9j6 krb5kdc[920](info): commencing operation
krb5kdc: starting...
W20260620 14:47:14.760135 388 mini_kdc.cc:121] Time spent starting KDC: real 1.160s user 0.004s sys 0.011s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
W20260620 14:47:15.008978 388 mini_kdc.cc:253] Time spent creating user principal test-user: real 0.186s user 0.002s sys 0.002s
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
W20260620 14:47:15.120458 388 mini_kdc.cc:253] Time spent creating user principal joe-interloper: real 0.111s user 0.000s sys 0.004s
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskUci3DU/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781966829091300-388-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskUci3DU/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781966829091300-388-0/krb5kdc/test-user.keytab.
Jun 20 14:47:15 dist-test-slave-q9j6 krb5kdc[920](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781966835, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-20T14:47:15Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T14:47:15Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.97.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskUci3DU/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781966829091300-388-0/krb5kdc/kudu_127.0.97.62.keytab.
Entry for principal kudu/127.0.97.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskUci3DU/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781966829091300-388-0/krb5kdc/kudu_127.0.97.62.keytab.
I20260620 14:47:15.233232 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskUci3DU/build/asan/bin/kudu
/tmp/dist-test-taskUci3DU/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskUci3DU/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781966829091300-388-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskUci3DU/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781966829091300-388-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskUci3DU/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781966829091300-388-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskUci3DU/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781966829091300-388-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:40505
--webserver_interface=127.0.97.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskUci3DU/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781966829091300-388-0/krb5kdc/kudu_127.0.97.62.keytab
--principal=kudu/127.0.97.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:33353
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.97.62:40505
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskUci3DU/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781966829091300-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskUci3DU/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781966829091300-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskUci3DU/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781966829091300-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 14:47:15.577551 933 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 14:47:15.577873 933 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 14:47:15.577962 933 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 14:47:15.585292 933 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 14:47:15.585417 933 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 14:47:15.585464 933 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 14:47:15.585517 933 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 14:47:15.585553 933 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 14:47:15.595327 933 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:33353
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskUci3DU/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781966829091300-388-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskUci3DU/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781966829091300-388-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.97.62:40505
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskUci3DU/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781966829091300-388-0/krb5kdc/kudu_127.0.97.62.keytab
--principal=kudu/127.0.97.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:40505
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskUci3DU/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781966829091300-388-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.97.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskUci3DU/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781966829091300-388-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 13:41:20 UTC on ccycloud.quasar-vdagoi.root.comops.site
build id 4457462
ASAN enabled
I20260620 14:47:15.596715 933 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 14:47:15.597990 933 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 14:47:15.609884 939 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 14:47:15.609935 941 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 14:47:15.609854 938 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 14:47:15.610919 933 server_base.cc:1034] running on GCE node
I20260620 14:47:15.611847 933 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 14:47:15.613421 933 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 14:47:15.614660 933 hybrid_clock.cc:648] HybridClock initialized: now 1781966835614613 us; error 59 us; skew 500 ppm
I20260620 14:47:15.615016 933 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 14:47:15 dist-test-slave-q9j6 krb5kdc[920](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781966835, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 14:47:15.618109 933 init.cc:377] Logged in from keytab as kudu/127.0.97.62@KRBTEST.COM (short username kudu)
I20260620 14:47:15.619706 933 webserver.cc:460] Webserver started at http://127.0.97.62:34487/ using document root <none> and password file <none>
I20260620 14:47:15.620524 933 fs_manager.cc:359] Metadata directory not provided
I20260620 14:47:15.620673 933 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 14:47:15.620990 933 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 14:47:15.624038 933 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskUci3DU/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781966829091300-388-0/minicluster-data/master-0/data/instance:
uuid: "9fbd52e0ed484ec18256926bc9cfb992"
format_stamp: "Formatted at 2026-06-20 14:47:15 on dist-test-slave-q9j6"
server_key: "1d1efdf7ab3cdf4eb661f845f771776d"
server_key_iv: "50f1dec5babe860fbb4123cc12f1955f"
server_key_version: "encryptionkey@0"
I20260620 14:47:15.624815 933 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskUci3DU/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781966829091300-388-0/minicluster-data/master-0/wal/instance:
uuid: "9fbd52e0ed484ec18256926bc9cfb992"
format_stamp: "Formatted at 2026-06-20 14:47:15 on dist-test-slave-q9j6"
server_key: "1d1efdf7ab3cdf4eb661f845f771776d"
server_key_iv: "50f1dec5babe860fbb4123cc12f1955f"
server_key_version: "encryptionkey@0"
I20260620 14:47:15.630257 933 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260620 14:47:15.633811 948 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 14:47:15.635176 933 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260620 14:47:15.635375 933 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskUci3DU/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781966829091300-388-0/minicluster-data/master-0/data,/tmp/dist-test-taskUci3DU/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781966829091300-388-0/minicluster-data/master-0/wal
uuid: "9fbd52e0ed484ec18256926bc9cfb992"
format_stamp: "Formatted at 2026-06-20 14:47:15 on dist-test-slave-q9j6"
server_key: "1d1efdf7ab3cdf4eb661f845f771776d"
server_key_iv: "50f1dec5babe860fbb4123cc12f1955f"
server_key_version: "encryptionkey@0"
I20260620 14:47:15.635583 933 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskUci3DU/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781966829091300-388-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskUci3DU/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781966829091300-388-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskUci3DU/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781966829091300-388-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 14:47:15.651139 933 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 14:47:15.655205 933 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 14:47:15.655512 933 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 14:47:15.674527 933 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.62:40505
I20260620 14:47:15.674584 999 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.62:40505 every 8 connection(s)
I20260620 14:47:15.676399 933 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskUci3DU/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781966829091300-388-0/minicluster-data/master-0/data/info.pb
I20260620 14:47:15.680471 1000 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 14:47:15.681599 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskUci3DU/build/asan/bin/kudu as pid 933
I20260620 14:47:15.681823 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskUci3DU/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781966829091300-388-0/minicluster-data/master-0/wal/instance
I20260620 14:47:15.682493 388 external_mini_cluster.cc:1428] Setting key 3734d7dd8116f5649c4bd26fdd5b5d47
I20260620 14:47:15.695156 1000 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 9fbd52e0ed484ec18256926bc9cfb992: Bootstrap starting.
Jun 20 14:47:15 dist-test-slave-q9j6 krb5kdc[920](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781966835, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260620 14:47:15.698928 1000 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 9fbd52e0ed484ec18256926bc9cfb992: Neither blocks nor log segments found. Creating new log.
I20260620 14:47:15.700273 1000 log.cc:824] T 00000000000000000000000000000000 P 9fbd52e0ed484ec18256926bc9cfb992: Log is configured to *not* fsync() on all Append() calls
I20260620 14:47:15.703265 1000 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 9fbd52e0ed484ec18256926bc9cfb992: No bootstrap required, opened a new log
I20260620 14:47:15.706010 1003 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:47:15.684541 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59282 (local address 127.0.97.62:40505)
0620 14:47:15.685343 (+ 802us) server_negotiation.cc:206] Beginning negotiation
0620 14:47:15.685358 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0620 14:47:15.685405 (+ 47us) server_negotiation.cc:407] Connection header received
0620 14:47:15.687013 (+ 1608us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:47:15.687043 (+ 30us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:47:15.687483 (+ 440us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:47:15.687934 (+ 451us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:47:15.689670 (+ 1736us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:47:15.691000 (+ 1330us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:47:15.692797 (+ 1797us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:47:15.693492 (+ 695us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:47:15.693623 (+ 131us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:47:15.698340 (+ 4717us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 14:47:15.698349 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 14:47:15.698365 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 14:47:15.698411 (+ 46us) server_negotiation.cc:922] calling sasl_server_start()
0620 14:47:15.701674 (+ 3263us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:47:15.702419 (+ 745us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:47:15.702425 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:47:15.702432 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:47:15.702593 (+ 161us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:47:15.702996 (+ 403us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:47:15.703000 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:47:15.703003 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:47:15.703591 (+ 588us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 14:47:15.703964 (+ 373us) server_negotiation.cc:1035] Waiting for connection context
0620 14:47:15.704466 (+ 502us) server_negotiation.cc:299] Negotiation successful
0620 14:47:15.704906 (+ 440us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":4,"server-negotiator.queue_time_us":584,"thread_start_us":374,"threads_started":1}
I20260620 14:47:15.711388 1000 raft_consensus.cc:348] T 00000000000000000000000000000000 P 9fbd52e0ed484ec18256926bc9cfb992 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9fbd52e0ed484ec18256926bc9cfb992" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 40505 } }
I20260620 14:47:15.711879 1000 raft_consensus.cc:374] T 00000000000000000000000000000000 P 9fbd52e0ed484ec18256926bc9cfb992 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 14:47:15.711989 1000 raft_consensus.cc:729] T 00000000000000000000000000000000 P 9fbd52e0ed484ec18256926bc9cfb992 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9fbd52e0ed484ec18256926bc9cfb992, State: Initialized, Role: FOLLOWER
I20260620 14:47:15.712672 1000 consensus_queue.cc:260] T 00000000000000000000000000000000 P 9fbd52e0ed484ec18256926bc9cfb992 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9fbd52e0ed484ec18256926bc9cfb992" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 40505 } }
I20260620 14:47:15.712920 1000 raft_consensus.cc:388] T 00000000000000000000000000000000 P 9fbd52e0ed484ec18256926bc9cfb992 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 14:47:15.713074 1000 raft_consensus.cc:482] T 00000000000000000000000000000000 P 9fbd52e0ed484ec18256926bc9cfb992 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 14:47:15.713274 1000 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 9fbd52e0ed484ec18256926bc9cfb992 [term 0 FOLLOWER]: Advancing to term 1
I20260620 14:47:15.715967 1000 raft_consensus.cc:504] T 00000000000000000000000000000000 P 9fbd52e0ed484ec18256926bc9cfb992 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9fbd52e0ed484ec18256926bc9cfb992" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 40505 } }
I20260620 14:47:15.716516 1000 leader_election.cc:302] T 00000000000000000000000000000000 P 9fbd52e0ed484ec18256926bc9cfb992 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 9fbd52e0ed484ec18256926bc9cfb992; no voters:
I20260620 14:47:15.717099 1000 leader_election.cc:288] T 00000000000000000000000000000000 P 9fbd52e0ed484ec18256926bc9cfb992 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260620 14:47:15.717238 1005 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 9fbd52e0ed484ec18256926bc9cfb992 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 14:47:15.718062 1005 raft_consensus.cc:686] T 00000000000000000000000000000000 P 9fbd52e0ed484ec18256926bc9cfb992 [term 1 LEADER]: Becoming Leader. State: Replica: 9fbd52e0ed484ec18256926bc9cfb992, State: Running, Role: LEADER
I20260620 14:47:15.718639 1005 consensus_queue.cc:237] T 00000000000000000000000000000000 P 9fbd52e0ed484ec18256926bc9cfb992 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9fbd52e0ed484ec18256926bc9cfb992" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 40505 } }
I20260620 14:47:15.719703 1000 sys_catalog.cc:564] T 00000000000000000000000000000000 P 9fbd52e0ed484ec18256926bc9cfb992 [sys.catalog]: configured and running, proceeding with master startup.
I20260620 14:47:15.721921 1006 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9fbd52e0ed484ec18256926bc9cfb992 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "9fbd52e0ed484ec18256926bc9cfb992" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9fbd52e0ed484ec18256926bc9cfb992" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 40505 } } }
I20260620 14:47:15.721845 1007 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9fbd52e0ed484ec18256926bc9cfb992 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 9fbd52e0ed484ec18256926bc9cfb992. Latest consensus state: current_term: 1 leader_uuid: "9fbd52e0ed484ec18256926bc9cfb992" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9fbd52e0ed484ec18256926bc9cfb992" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 40505 } } }
I20260620 14:47:15.722340 1006 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9fbd52e0ed484ec18256926bc9cfb992 [sys.catalog]: This master's current role is: LEADER
I20260620 14:47:15.722348 1007 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9fbd52e0ed484ec18256926bc9cfb992 [sys.catalog]: This master's current role is: LEADER
I20260620 14:47:15.724138 1012 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 14:47:15.729596 1012 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 14:47:15.739621 1012 catalog_manager.cc:1347] Generated new cluster ID: ad30a028b85b4fa9b8245630e5687929
I20260620 14:47:15.739796 1012 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 14:47:15.751324 1012 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 14:47:15.753077 1012 catalog_manager.cc:1504] Loading token signing keys...
I20260620 14:47:15.766722 1012 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 9fbd52e0ed484ec18256926bc9cfb992: Generated new TSK 0
I20260620 14:47:15.767644 1012 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.0.97.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskUci3DU/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781966829091300-388-0/krb5kdc/kudu_127.0.97.1.keytab.
Entry for principal kudu/127.0.97.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskUci3DU/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781966829091300-388-0/krb5kdc/kudu_127.0.97.1.keytab.
I20260620 14:47:15.819550 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskUci3DU/build/asan/bin/kudu
/tmp/dist-test-taskUci3DU/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskUci3DU/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781966829091300-388-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskUci3DU/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781966829091300-388-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskUci3DU/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781966829091300-388-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskUci3DU/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781966829091300-388-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.1:0
--local_ip_for_outbound_sockets=127.0.97.1
--webserver_interface=127.0.97.1
--webserver_port=0
--tserver_master_addrs=127.0.97.62:40505
--keytab_file=/tmp/dist-test-taskUci3DU/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781966829091300-388-0/krb5kdc/kudu_127.0.97.1.keytab
--principal=kudu/127.0.97.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:33353
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-taskUci3DU/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781966829091300-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskUci3DU/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781966829091300-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskUci3DU/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781966829091300-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 14:47:16.169624 1026 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 14:47:16.169926 1026 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 14:47:16.170094 1026 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 14:47:16.177435 1026 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 14:47:16.177558 1026 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 14:47:16.177757 1026 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.1
I20260620 14:47:16.187772 1026 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:33353
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskUci3DU/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781966829091300-388-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskUci3DU/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781966829091300-388-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskUci3DU/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781966829091300-388-0/krb5kdc/kudu_127.0.97.1.keytab
--principal=kudu/127.0.97.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskUci3DU/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781966829091300-388-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.97.1
--webserver_port=0
--tserver_master_addrs=127.0.97.62:40505
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.1
--log_dir=/tmp/dist-test-taskUci3DU/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781966829091300-388-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 13:41:20 UTC on ccycloud.quasar-vdagoi.root.comops.site
build id 4457462
ASAN enabled
I20260620 14:47:16.189103 1026 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 14:47:16.190397 1026 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 14:47:16.203155 1034 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 14:47:16.203255 1032 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 14:47:16.203210 1031 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 14:47:16.209345 1026 server_base.cc:1034] running on GCE node
I20260620 14:47:16.210007 1026 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 14:47:16.211036 1026 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 14:47:16.212263 1026 hybrid_clock.cc:648] HybridClock initialized: now 1781966836212219 us; error 63 us; skew 500 ppm
I20260620 14:47:16.212597 1026 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 14:47:16 dist-test-slave-q9j6 krb5kdc[920](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781966836, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 14:47:16.215955 1026 init.cc:377] Logged in from keytab as kudu/127.0.97.1@KRBTEST.COM (short username kudu)
I20260620 14:47:16.217573 1026 webserver.cc:460] Webserver started at http://127.0.97.1:33143/ using document root <none> and password file <none>
I20260620 14:47:16.218345 1026 fs_manager.cc:359] Metadata directory not provided
I20260620 14:47:16.218492 1026 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 14:47:16.218851 1026 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 14:47:16.222149 1026 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskUci3DU/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781966829091300-388-0/minicluster-data/ts-0/data/instance:
uuid: "8fda64b6e2f44e8a9fa1e54a1575f881"
format_stamp: "Formatted at 2026-06-20 14:47:16 on dist-test-slave-q9j6"
server_key: "332bddfe93b0cf6ce61d453521d42b03"
server_key_iv: "a77dc209674c650f8a687103e2176a53"
server_key_version: "encryptionkey@0"
I20260620 14:47:16.223013 1026 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskUci3DU/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781966829091300-388-0/minicluster-data/ts-0/wal/instance:
uuid: "8fda64b6e2f44e8a9fa1e54a1575f881"
format_stamp: "Formatted at 2026-06-20 14:47:16 on dist-test-slave-q9j6"
server_key: "332bddfe93b0cf6ce61d453521d42b03"
server_key_iv: "a77dc209674c650f8a687103e2176a53"
server_key_version: "encryptionkey@0"
I20260620 14:47:16.228730 1026 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.007s sys 0.001s
I20260620 14:47:16.232334 1041 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 14:47:16.233865 1026 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260620 14:47:16.234050 1026 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskUci3DU/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781966829091300-388-0/minicluster-data/ts-0/data,/tmp/dist-test-taskUci3DU/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781966829091300-388-0/minicluster-data/ts-0/wal
uuid: "8fda64b6e2f44e8a9fa1e54a1575f881"
format_stamp: "Formatted at 2026-06-20 14:47:16 on dist-test-slave-q9j6"
server_key: "332bddfe93b0cf6ce61d453521d42b03"
server_key_iv: "a77dc209674c650f8a687103e2176a53"
server_key_version: "encryptionkey@0"
I20260620 14:47:16.234272 1026 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskUci3DU/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781966829091300-388-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskUci3DU/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781966829091300-388-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskUci3DU/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781966829091300-388-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 14:47:16.248903 1026 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 14:47:16.253705 1026 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 14:47:16.254022 1026 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 14:47:16.255014 1026 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 14:47:16.256421 1026 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 14:47:16.256528 1026 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 14:47:16.256686 1026 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 14:47:16.256764 1026 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 14:47:16.290683 1026 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.1:46737
I20260620 14:47:16.290731 1153 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.1:46737 every 8 connection(s)
I20260620 14:47:16.292047 1026 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskUci3DU/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781966829091300-388-0/minicluster-data/ts-0/data/info.pb
I20260620 14:47:16.300498 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskUci3DU/build/asan/bin/kudu as pid 1026
I20260620 14:47:16.300722 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskUci3DU/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781966829091300-388-0/minicluster-data/ts-0/wal/instance
I20260620 14:47:16.301491 388 external_mini_cluster.cc:1428] Setting key 1901f7d4b99ae546cc376f1f0bfe0129
Jun 20 14:47:16 dist-test-slave-q9j6 krb5kdc[920](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781966836, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.1@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260620 14:47:16.313902 1156 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:47:16.294594 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.1:57777 (local address 127.0.97.62:40505)
0620 14:47:16.295029 (+ 435us) server_negotiation.cc:206] Beginning negotiation
0620 14:47:16.295036 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0620 14:47:16.296084 (+ 1048us) server_negotiation.cc:407] Connection header received
0620 14:47:16.297843 (+ 1759us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:47:16.297848 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:47:16.297926 (+ 78us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:47:16.298024 (+ 98us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:47:16.299592 (+ 1568us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:47:16.300395 (+ 803us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:47:16.302778 (+ 2383us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:47:16.303421 (+ 643us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:47:16.303559 (+ 138us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:47:16.307974 (+ 4415us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 14:47:16.307980 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 14:47:16.307983 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 14:47:16.308017 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0620 14:47:16.310602 (+ 2585us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:47:16.311424 (+ 822us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:47:16.311434 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:47:16.311436 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:47:16.311555 (+ 119us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:47:16.312082 (+ 527us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:47:16.312089 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:47:16.312096 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:47:16.312380 (+ 284us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 14:47:16.312540 (+ 160us) server_negotiation.cc:1035] Waiting for connection context
0620 14:47:16.313285 (+ 745us) server_negotiation.cc:299] Negotiation successful
0620 14:47:16.313520 (+ 235us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":313,"thread_start_us":207,"threads_started":1}
I20260620 14:47:16.315064 1157 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:47:16.295218 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:40505 (local address 127.0.97.1:57777)
0620 14:47:16.295928 (+ 710us) negotiation.cc:107] Waiting for socket to connect
0620 14:47:16.295968 (+ 40us) client_negotiation.cc:174] Beginning negotiation
0620 14:47:16.297582 (+ 1614us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 14:47:16.298249 (+ 667us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 14:47:16.298257 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 14:47:16.298773 (+ 516us) client_negotiation.cc:189] Negotiated authn=SASL
0620 14:47:16.299416 (+ 643us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:47:16.299433 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:47:16.300574 (+ 1141us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:47:16.300581 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:47:16.302630 (+ 2049us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:47:16.302643 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:47:16.303717 (+ 1074us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:47:16.303725 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:47:16.304052 (+ 327us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:47:16.304824 (+ 772us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 14:47:16.304861 (+ 37us) client_negotiation.cc:656] calling sasl_client_start()
0620 14:47:16.307725 (+ 2864us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 14:47:16.310788 (+ 3063us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 14:47:16.310797 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 14:47:16.310817 (+ 20us) client_negotiation.cc:760] calling sasl_client_step()
0620 14:47:16.311280 (+ 463us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 14:47:16.311739 (+ 459us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 14:47:16.311744 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 14:47:16.311748 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0620 14:47:16.311948 (+ 200us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 14:47:16.312517 (+ 569us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 14:47:16.312526 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 14:47:16.312899 (+ 373us) client_negotiation.cc:769] Sending connection context
0620 14:47:16.313189 (+ 290us) client_negotiation.cc:240] Negotiation successful
0620 14:47:16.313764 (+ 575us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":487,"thread_start_us":315,"threads_started":1}
I20260620 14:47:16.316720 1154 heartbeater.cc:347] Connected to a master server at 127.0.97.62:40505
I20260620 14:47:16.317091 1154 heartbeater.cc:464] Registering TS with master...
I20260620 14:47:16.318056 1154 heartbeater.cc:511] Master 127.0.97.62:40505 requested a full tablet report, sending...
WARNING: no policy specified for kudu/127.0.97.2@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.2@KRBTEST.COM" created.
I20260620 14:47:16.320564 965 ts_manager.cc:194] Registered new tserver with Master: 8fda64b6e2f44e8a9fa1e54a1575f881 (127.0.97.1:46737)
I20260620 14:47:16.322091 965 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.97.1@KRBTEST.COM'} at 127.0.97.1:57777
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskUci3DU/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781966829091300-388-0/krb5kdc/kudu_127.0.97.2.keytab.
Entry for principal kudu/127.0.97.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskUci3DU/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781966829091300-388-0/krb5kdc/kudu_127.0.97.2.keytab.
I20260620 14:47:16.340865 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskUci3DU/build/asan/bin/kudu
/tmp/dist-test-taskUci3DU/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskUci3DU/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781966829091300-388-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskUci3DU/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781966829091300-388-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskUci3DU/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781966829091300-388-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskUci3DU/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781966829091300-388-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.2:0
--local_ip_for_outbound_sockets=127.0.97.2
--webserver_interface=127.0.97.2
--webserver_port=0
--tserver_master_addrs=127.0.97.62:40505
--keytab_file=/tmp/dist-test-taskUci3DU/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781966829091300-388-0/krb5kdc/kudu_127.0.97.2.keytab
--principal=kudu/127.0.97.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:33353
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-taskUci3DU/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781966829091300-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskUci3DU/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781966829091300-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskUci3DU/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781966829091300-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 14:47:16.668651 1160 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 14:47:16.668951 1160 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 14:47:16.669045 1160 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 14:47:16.676483 1160 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 14:47:16.676590 1160 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 14:47:16.676764 1160 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.2
I20260620 14:47:16.686664 1160 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:33353
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskUci3DU/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781966829091300-388-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskUci3DU/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781966829091300-388-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskUci3DU/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781966829091300-388-0/krb5kdc/kudu_127.0.97.2.keytab
--principal=kudu/127.0.97.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskUci3DU/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781966829091300-388-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.97.2
--webserver_port=0
--tserver_master_addrs=127.0.97.62:40505
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.2
--log_dir=/tmp/dist-test-taskUci3DU/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781966829091300-388-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 13:41:20 UTC on ccycloud.quasar-vdagoi.root.comops.site
build id 4457462
ASAN enabled
I20260620 14:47:16.688006 1160 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 14:47:16.689342 1160 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 14:47:16.702394 1166 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 14:47:16.702334 1168 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 14:47:16.702370 1165 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 14:47:16.703130 1160 server_base.cc:1034] running on GCE node
I20260620 14:47:16.703873 1160 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 14:47:16.704840 1160 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 14:47:16.706072 1160 hybrid_clock.cc:648] HybridClock initialized: now 1781966836706020 us; error 49 us; skew 500 ppm
I20260620 14:47:16.706411 1160 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 14:47:16 dist-test-slave-q9j6 krb5kdc[920](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781966836, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 14:47:16.709578 1160 init.cc:377] Logged in from keytab as kudu/127.0.97.2@KRBTEST.COM (short username kudu)
I20260620 14:47:16.711139 1160 webserver.cc:460] Webserver started at http://127.0.97.2:42263/ using document root <none> and password file <none>
I20260620 14:47:16.711932 1160 fs_manager.cc:359] Metadata directory not provided
I20260620 14:47:16.712078 1160 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 14:47:16.712407 1160 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 14:47:16.715332 1160 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskUci3DU/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781966829091300-388-0/minicluster-data/ts-1/data/instance:
uuid: "e122103b272f4a58809f540448f227ff"
format_stamp: "Formatted at 2026-06-20 14:47:16 on dist-test-slave-q9j6"
server_key: "9b1110d8faaf2341bd2206655610096b"
server_key_iv: "5e385024c3b4777bec89706063a0d66d"
server_key_version: "encryptionkey@0"
I20260620 14:47:16.716208 1160 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskUci3DU/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781966829091300-388-0/minicluster-data/ts-1/wal/instance:
uuid: "e122103b272f4a58809f540448f227ff"
format_stamp: "Formatted at 2026-06-20 14:47:16 on dist-test-slave-q9j6"
server_key: "9b1110d8faaf2341bd2206655610096b"
server_key_iv: "5e385024c3b4777bec89706063a0d66d"
server_key_version: "encryptionkey@0"
I20260620 14:47:16.721717 1160 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260620 14:47:16.725471 1175 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 14:47:16.726892 1160 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260620 14:47:16.727080 1160 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskUci3DU/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781966829091300-388-0/minicluster-data/ts-1/data,/tmp/dist-test-taskUci3DU/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781966829091300-388-0/minicluster-data/ts-1/wal
uuid: "e122103b272f4a58809f540448f227ff"
format_stamp: "Formatted at 2026-06-20 14:47:16 on dist-test-slave-q9j6"
server_key: "9b1110d8faaf2341bd2206655610096b"
server_key_iv: "5e385024c3b4777bec89706063a0d66d"
server_key_version: "encryptionkey@0"
I20260620 14:47:16.727304 1160 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskUci3DU/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781966829091300-388-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskUci3DU/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781966829091300-388-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskUci3DU/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781966829091300-388-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 14:47:16.754364 1160 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 14:47:16.758636 1160 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 14:47:16.758960 1160 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 14:47:16.759968 1160 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 14:47:16.761346 1160 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 14:47:16.761452 1160 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 14:47:16.761600 1160 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 14:47:16.761672 1160 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 14:47:16.794277 1160 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.2:46475
I20260620 14:47:16.794332 1287 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.2:46475 every 8 connection(s)
I20260620 14:47:16.795622 1160 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskUci3DU/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781966829091300-388-0/minicluster-data/ts-1/data/info.pb
I20260620 14:47:16.798606 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskUci3DU/build/asan/bin/kudu as pid 1160
I20260620 14:47:16.798825 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskUci3DU/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781966829091300-388-0/minicluster-data/ts-1/wal/instance
I20260620 14:47:16.799237 388 external_mini_cluster.cc:1428] Setting key b13b3af2d085096b97082c4f7c3a2341
Jun 20 14:47:16 dist-test-slave-q9j6 krb5kdc[920](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781966836, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.2@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
WARNING: no policy specified for kudu/127.0.97.3@KRBTEST.COM; defaulting to no policy
I20260620 14:47:16.818097 1156 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:47:16.798209 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.2:34863 (local address 127.0.97.62:40505)
0620 14:47:16.798417 (+ 208us) server_negotiation.cc:206] Beginning negotiation
0620 14:47:16.798423 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 14:47:16.799918 (+ 1495us) server_negotiation.cc:407] Connection header received
0620 14:47:16.801768 (+ 1850us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:47:16.801774 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:47:16.801855 (+ 81us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:47:16.801983 (+ 128us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:47:16.803526 (+ 1543us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:47:16.804710 (+ 1184us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:47:16.806342 (+ 1632us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:47:16.806860 (+ 518us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:47:16.807034 (+ 174us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:47:16.811133 (+ 4099us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 14:47:16.811141 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 14:47:16.811144 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 14:47:16.811179 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0620 14:47:16.814732 (+ 3553us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:47:16.815580 (+ 848us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:47:16.815586 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:47:16.815591 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:47:16.815813 (+ 222us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:47:16.816375 (+ 562us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:47:16.816380 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:47:16.816384 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:47:16.816787 (+ 403us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 14:47:16.816928 (+ 141us) server_negotiation.cc:1035] Waiting for connection context
0620 14:47:16.817678 (+ 750us) server_negotiation.cc:299] Negotiation successful
0620 14:47:16.817860 (+ 182us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":85}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.3@KRBTEST.COM" created.
I20260620 14:47:16.819418 1290 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:47:16.798840 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:40505 (local address 127.0.97.2:34863)
0620 14:47:16.799732 (+ 892us) negotiation.cc:107] Waiting for socket to connect
0620 14:47:16.799778 (+ 46us) client_negotiation.cc:174] Beginning negotiation
0620 14:47:16.801486 (+ 1708us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 14:47:16.802199 (+ 713us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 14:47:16.802208 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 14:47:16.802745 (+ 537us) client_negotiation.cc:189] Negotiated authn=SASL
0620 14:47:16.803332 (+ 587us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:47:16.803350 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:47:16.804871 (+ 1521us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:47:16.804875 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:47:16.806155 (+ 1280us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:47:16.806167 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:47:16.807133 (+ 966us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:47:16.807151 (+ 18us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:47:16.807401 (+ 250us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:47:16.808189 (+ 788us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 14:47:16.808213 (+ 24us) client_negotiation.cc:656] calling sasl_client_start()
0620 14:47:16.810832 (+ 2619us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 14:47:16.814959 (+ 4127us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 14:47:16.814968 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 14:47:16.814982 (+ 14us) client_negotiation.cc:760] calling sasl_client_step()
0620 14:47:16.815436 (+ 454us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 14:47:16.815949 (+ 513us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 14:47:16.815954 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 14:47:16.815957 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0620 14:47:16.816250 (+ 293us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 14:47:16.816925 (+ 675us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 14:47:16.816931 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 14:47:16.817294 (+ 363us) client_negotiation.cc:769] Sending connection context
0620 14:47:16.817540 (+ 246us) client_negotiation.cc:240] Negotiation successful
0620 14:47:16.818023 (+ 483us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":584,"thread_start_us":348,"threads_started":1}
I20260620 14:47:16.821079 1288 heartbeater.cc:347] Connected to a master server at 127.0.97.62:40505
I20260620 14:47:16.821444 1288 heartbeater.cc:464] Registering TS with master...
I20260620 14:47:16.822397 1288 heartbeater.cc:511] Master 127.0.97.62:40505 requested a full tablet report, sending...
I20260620 14:47:16.824170 965 ts_manager.cc:194] Registered new tserver with Master: e122103b272f4a58809f540448f227ff (127.0.97.2:46475)
I20260620 14:47:16.825455 965 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.97.2@KRBTEST.COM'} at 127.0.97.2:34863
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskUci3DU/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781966829091300-388-0/krb5kdc/kudu_127.0.97.3.keytab.
Entry for principal kudu/127.0.97.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskUci3DU/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781966829091300-388-0/krb5kdc/kudu_127.0.97.3.keytab.
I20260620 14:47:16.841189 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskUci3DU/build/asan/bin/kudu
/tmp/dist-test-taskUci3DU/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskUci3DU/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781966829091300-388-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskUci3DU/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781966829091300-388-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskUci3DU/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781966829091300-388-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskUci3DU/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781966829091300-388-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.3:0
--local_ip_for_outbound_sockets=127.0.97.3
--webserver_interface=127.0.97.3
--webserver_port=0
--tserver_master_addrs=127.0.97.62:40505
--keytab_file=/tmp/dist-test-taskUci3DU/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781966829091300-388-0/krb5kdc/kudu_127.0.97.3.keytab
--principal=kudu/127.0.97.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:33353
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-taskUci3DU/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781966829091300-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskUci3DU/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781966829091300-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskUci3DU/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781966829091300-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 14:47:17.172616 1293 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 14:47:17.173010 1293 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 14:47:17.173195 1293 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 14:47:17.180766 1293 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 14:47:17.180939 1293 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 14:47:17.181133 1293 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.3
I20260620 14:47:17.191156 1293 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:33353
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskUci3DU/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781966829091300-388-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskUci3DU/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781966829091300-388-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskUci3DU/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781966829091300-388-0/krb5kdc/kudu_127.0.97.3.keytab
--principal=kudu/127.0.97.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskUci3DU/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781966829091300-388-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.97.3
--webserver_port=0
--tserver_master_addrs=127.0.97.62:40505
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.3
--log_dir=/tmp/dist-test-taskUci3DU/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781966829091300-388-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 13:41:20 UTC on ccycloud.quasar-vdagoi.root.comops.site
build id 4457462
ASAN enabled
I20260620 14:47:17.192582 1293 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 14:47:17.193936 1293 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 14:47:17.207235 1298 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 14:47:17.207214 1301 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 14:47:17.207437 1299 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 14:47:17.213079 1293 server_base.cc:1034] running on GCE node
I20260620 14:47:17.213805 1293 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 14:47:17.214840 1293 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 14:47:17.216079 1293 hybrid_clock.cc:648] HybridClock initialized: now 1781966837216008 us; error 69 us; skew 500 ppm
I20260620 14:47:17.216481 1293 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 14:47:17 dist-test-slave-q9j6 krb5kdc[920](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781966837, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 14:47:17.219728 1293 init.cc:377] Logged in from keytab as kudu/127.0.97.3@KRBTEST.COM (short username kudu)
I20260620 14:47:17.221419 1293 webserver.cc:460] Webserver started at http://127.0.97.3:43683/ using document root <none> and password file <none>
I20260620 14:47:17.222137 1293 fs_manager.cc:359] Metadata directory not provided
I20260620 14:47:17.222239 1293 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 14:47:17.222500 1293 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 14:47:17.225590 1293 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskUci3DU/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781966829091300-388-0/minicluster-data/ts-2/data/instance:
uuid: "d34fa4445c8e45918b1ecd8007e551a8"
format_stamp: "Formatted at 2026-06-20 14:47:17 on dist-test-slave-q9j6"
server_key: "9f501801e1e94ec9b6294c5d12759442"
server_key_iv: "dc3f1c8c2f666a86e5a50758a108f8b1"
server_key_version: "encryptionkey@0"
I20260620 14:47:17.226359 1293 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskUci3DU/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781966829091300-388-0/minicluster-data/ts-2/wal/instance:
uuid: "d34fa4445c8e45918b1ecd8007e551a8"
format_stamp: "Formatted at 2026-06-20 14:47:17 on dist-test-slave-q9j6"
server_key: "9f501801e1e94ec9b6294c5d12759442"
server_key_iv: "dc3f1c8c2f666a86e5a50758a108f8b1"
server_key_version: "encryptionkey@0"
I20260620 14:47:17.232190 1293 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.003s sys 0.004s
I20260620 14:47:17.235999 1308 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 14:47:17.237452 1293 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.001s
I20260620 14:47:17.237625 1293 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskUci3DU/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781966829091300-388-0/minicluster-data/ts-2/data,/tmp/dist-test-taskUci3DU/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781966829091300-388-0/minicluster-data/ts-2/wal
uuid: "d34fa4445c8e45918b1ecd8007e551a8"
format_stamp: "Formatted at 2026-06-20 14:47:17 on dist-test-slave-q9j6"
server_key: "9f501801e1e94ec9b6294c5d12759442"
server_key_iv: "dc3f1c8c2f666a86e5a50758a108f8b1"
server_key_version: "encryptionkey@0"
I20260620 14:47:17.237838 1293 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskUci3DU/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781966829091300-388-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskUci3DU/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781966829091300-388-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskUci3DU/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781966829091300-388-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 14:47:17.254992 1293 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 14:47:17.259369 1293 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 14:47:17.259752 1293 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 14:47:17.260808 1293 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 14:47:17.262279 1293 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 14:47:17.262355 1293 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 14:47:17.262512 1293 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 14:47:17.262573 1293 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 14:47:17.296092 1293 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.3:35645
I20260620 14:47:17.296103 1420 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.3:35645 every 8 connection(s)
I20260620 14:47:17.297701 1293 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskUci3DU/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781966829091300-388-0/minicluster-data/ts-2/data/info.pb
I20260620 14:47:17.298272 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskUci3DU/build/asan/bin/kudu as pid 1293
I20260620 14:47:17.298485 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskUci3DU/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781966829091300-388-0/minicluster-data/ts-2/wal/instance
I20260620 14:47:17.299013 388 external_mini_cluster.cc:1428] Setting key b57a322bcbc364e39c036677385fbe68
Jun 20 14:47:17 dist-test-slave-q9j6 krb5kdc[920](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781966837, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.3@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260620 14:47:17.319080 1156 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:47:17.301725 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.3:39781 (local address 127.0.97.62:40505)
0620 14:47:17.301911 (+ 186us) server_negotiation.cc:206] Beginning negotiation
0620 14:47:17.301916 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 14:47:17.302352 (+ 436us) server_negotiation.cc:407] Connection header received
0620 14:47:17.304300 (+ 1948us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:47:17.304304 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:47:17.304370 (+ 66us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:47:17.304455 (+ 85us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:47:17.305974 (+ 1519us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:47:17.306751 (+ 777us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:47:17.308841 (+ 2090us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:47:17.309264 (+ 423us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:47:17.309374 (+ 110us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:47:17.313195 (+ 3821us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 14:47:17.313205 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 14:47:17.313212 (+ 7us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 14:47:17.313261 (+ 49us) server_negotiation.cc:922] calling sasl_server_start()
0620 14:47:17.316336 (+ 3075us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:47:17.317070 (+ 734us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:47:17.317075 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:47:17.317077 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:47:17.317208 (+ 131us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:47:17.317591 (+ 383us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:47:17.317595 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:47:17.317597 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:47:17.317860 (+ 263us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 14:47:17.317970 (+ 110us) server_negotiation.cc:1035] Waiting for connection context
0620 14:47:17.318666 (+ 696us) server_negotiation.cc:299] Negotiation successful
0620 14:47:17.318830 (+ 164us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":66}
I20260620 14:47:17.320269 1423 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:47:17.301264 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:40505 (local address 127.0.97.3:39781)
0620 14:47:17.302159 (+ 895us) negotiation.cc:107] Waiting for socket to connect
0620 14:47:17.302225 (+ 66us) client_negotiation.cc:174] Beginning negotiation
0620 14:47:17.304028 (+ 1803us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 14:47:17.304678 (+ 650us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 14:47:17.304687 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 14:47:17.305223 (+ 536us) client_negotiation.cc:189] Negotiated authn=SASL
0620 14:47:17.305796 (+ 573us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:47:17.305814 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:47:17.306891 (+ 1077us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:47:17.306894 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:47:17.308677 (+ 1783us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:47:17.308691 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:47:17.309464 (+ 773us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:47:17.309469 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:47:17.309674 (+ 205us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:47:17.310444 (+ 770us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 14:47:17.310468 (+ 24us) client_negotiation.cc:656] calling sasl_client_start()
0620 14:47:17.312842 (+ 2374us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 14:47:17.316471 (+ 3629us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 14:47:17.316479 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 14:47:17.316494 (+ 15us) client_negotiation.cc:760] calling sasl_client_step()
0620 14:47:17.316922 (+ 428us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 14:47:17.317301 (+ 379us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 14:47:17.317304 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 14:47:17.317306 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 14:47:17.317483 (+ 177us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 14:47:17.317966 (+ 483us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 14:47:17.317972 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 14:47:17.318315 (+ 343us) client_negotiation.cc:769] Sending connection context
0620 14:47:17.318545 (+ 230us) client_negotiation.cc:240] Negotiation successful
0620 14:47:17.318935 (+ 390us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":514,"spinlock_wait_cycles":20736,"thread_start_us":451,"threads_started":1}
I20260620 14:47:17.321888 1421 heartbeater.cc:347] Connected to a master server at 127.0.97.62:40505
I20260620 14:47:17.322172 1421 heartbeater.cc:464] Registering TS with master...
I20260620 14:47:17.322880 1421 heartbeater.cc:511] Master 127.0.97.62:40505 requested a full tablet report, sending...
I20260620 14:47:17.324472 965 ts_manager.cc:194] Registered new tserver with Master: d34fa4445c8e45918b1ecd8007e551a8 (127.0.97.3:35645)
I20260620 14:47:17.325343 965 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.97.3@KRBTEST.COM'} at 127.0.97.3:39781
I20260620 14:47:17.325729 1154 heartbeater.cc:503] Master 127.0.97.62:40505 was elected leader, sending a full tablet report...
I20260620 14:47:17.327546 388 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260620 14:47:17.347476 1156 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:47:17.334061 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59290 (local address 127.0.97.62:40505)
0620 14:47:17.334245 (+ 184us) server_negotiation.cc:206] Beginning negotiation
0620 14:47:17.334250 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 14:47:17.335240 (+ 990us) server_negotiation.cc:407] Connection header received
0620 14:47:17.335381 (+ 141us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:47:17.335385 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:47:17.335449 (+ 64us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:47:17.335546 (+ 97us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:47:17.337056 (+ 1510us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:47:17.337866 (+ 810us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:47:17.339353 (+ 1487us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:47:17.339823 (+ 470us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:47:17.339928 (+ 105us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:47:17.342312 (+ 2384us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 14:47:17.342317 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 14:47:17.342320 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 14:47:17.342351 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0620 14:47:17.345124 (+ 2773us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:47:17.345786 (+ 662us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:47:17.345790 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:47:17.345792 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:47:17.345908 (+ 116us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:47:17.346222 (+ 314us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:47:17.346225 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:47:17.346227 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:47:17.346521 (+ 294us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 14:47:17.346708 (+ 187us) server_negotiation.cc:1035] Waiting for connection context
0620 14:47:17.346967 (+ 259us) server_negotiation.cc:299] Negotiation successful
0620 14:47:17.347124 (+ 157us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":69}
I20260620 14:47:17.354434 964 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:59290:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260620 14:47:17.361814 964 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260620 14:47:17.398628 1434 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:47:17.374315 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.3:35645 (local address 127.0.0.1:38480)
0620 14:47:17.387489 (+ 13174us) negotiation.cc:107] Waiting for socket to connect
0620 14:47:17.387515 (+ 26us) client_negotiation.cc:174] Beginning negotiation
0620 14:47:17.387957 (+ 442us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 14:47:17.389845 (+ 1888us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 14:47:17.389853 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 14:47:17.389895 (+ 42us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 14:47:17.390409 (+ 514us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:47:17.390425 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:47:17.393301 (+ 2876us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:47:17.393307 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:47:17.395801 (+ 2494us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:47:17.395814 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:47:17.397556 (+ 1742us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:47:17.397563 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:47:17.397705 (+ 142us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:47:17.397764 (+ 59us) client_negotiation.cc:769] Sending connection context
0620 14:47:17.397873 (+ 109us) client_negotiation.cc:240] Negotiation successful
0620 14:47:17.398096 (+ 223us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":13005,"mutex_wait_us":31,"thread_start_us":13303,"threads_started":1}
I20260620 14:47:17.398751 1433 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:47:17.374283 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:46737 (local address 127.0.0.1:37556)
0620 14:47:17.384964 (+ 10681us) negotiation.cc:107] Waiting for socket to connect
0620 14:47:17.385042 (+ 78us) client_negotiation.cc:174] Beginning negotiation
0620 14:47:17.385312 (+ 270us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 14:47:17.390584 (+ 5272us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 14:47:17.390588 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 14:47:17.390613 (+ 25us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 14:47:17.391382 (+ 769us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:47:17.391398 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:47:17.392884 (+ 1486us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:47:17.392892 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:47:17.395444 (+ 2552us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:47:17.395455 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:47:17.397337 (+ 1882us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:47:17.397344 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:47:17.397569 (+ 225us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:47:17.397640 (+ 71us) client_negotiation.cc:769] Sending connection context
0620 14:47:17.398136 (+ 496us) client_negotiation.cc:240] Negotiation successful
0620 14:47:17.398277 (+ 141us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":10277,"mutex_wait_us":30,"thread_start_us":10104,"threads_started":1}
I20260620 14:47:17.399354 1438 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:47:17.389147 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.2:46475 (local address 127.0.0.1:37646)
0620 14:47:17.390365 (+ 1218us) negotiation.cc:107] Waiting for socket to connect
0620 14:47:17.390392 (+ 27us) client_negotiation.cc:174] Beginning negotiation
0620 14:47:17.390560 (+ 168us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 14:47:17.391784 (+ 1224us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 14:47:17.391789 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 14:47:17.391814 (+ 25us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 14:47:17.392190 (+ 376us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:47:17.392211 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:47:17.394005 (+ 1794us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:47:17.394009 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:47:17.396237 (+ 2228us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:47:17.396246 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:47:17.398095 (+ 1849us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:47:17.398106 (+ 11us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:47:17.398291 (+ 185us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:47:17.398318 (+ 27us) client_negotiation.cc:769] Sending connection context
0620 14:47:17.398377 (+ 59us) client_negotiation.cc:240] Negotiation successful
0620 14:47:17.398512 (+ 135us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":995,"mutex_wait_us":3,"thread_start_us":1490,"threads_started":1}
I20260620 14:47:17.399963 1436 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:47:17.375040 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37556 (local address 127.0.97.1:46737)
0620 14:47:17.389810 (+ 14770us) server_negotiation.cc:206] Beginning negotiation
0620 14:47:17.389818 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0620 14:47:17.389846 (+ 28us) server_negotiation.cc:407] Connection header received
0620 14:47:17.389934 (+ 88us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:47:17.389941 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:47:17.390213 (+ 272us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:47:17.390439 (+ 226us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 14:47:17.391614 (+ 1175us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:47:17.392741 (+ 1127us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:47:17.395626 (+ 2885us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:47:17.397123 (+ 1497us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:47:17.397277 (+ 154us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:47:17.397556 (+ 279us) server_negotiation.cc:1035] Waiting for connection context
0620 14:47:17.399390 (+ 1834us) server_negotiation.cc:299] Negotiation successful
0620 14:47:17.399592 (+ 202us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":14576,"thread_start_us":14409,"threads_started":1}
I20260620 14:47:17.400128 1435 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:47:17.374363 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38480 (local address 127.0.97.3:35645)
0620 14:47:17.389027 (+ 14664us) server_negotiation.cc:206] Beginning negotiation
0620 14:47:17.389037 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0620 14:47:17.389076 (+ 39us) server_negotiation.cc:407] Connection header received
0620 14:47:17.389193 (+ 117us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:47:17.389203 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:47:17.389532 (+ 329us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:47:17.391172 (+ 1640us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 14:47:17.391444 (+ 272us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:47:17.393159 (+ 1715us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:47:17.395975 (+ 2816us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:47:17.397335 (+ 1360us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:47:17.398917 (+ 1582us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:47:17.399166 (+ 249us) server_negotiation.cc:1035] Waiting for connection context
0620 14:47:17.399266 (+ 100us) server_negotiation.cc:299] Negotiation successful
0620 14:47:17.399837 (+ 571us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":14482,"thread_start_us":14354,"threads_started":1}
I20260620 14:47:17.400442 1437 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:47:17.386016 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37646 (local address 127.0.97.2:46475)
0620 14:47:17.391134 (+ 5118us) server_negotiation.cc:206] Beginning negotiation
0620 14:47:17.391143 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0620 14:47:17.391171 (+ 28us) server_negotiation.cc:407] Connection header received
0620 14:47:17.391280 (+ 109us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:47:17.391291 (+ 11us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:47:17.391532 (+ 241us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:47:17.392383 (+ 851us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 14:47:17.392612 (+ 229us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:47:17.393850 (+ 1238us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:47:17.396384 (+ 2534us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:47:17.397426 (+ 1042us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:47:17.399660 (+ 2234us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:47:17.399864 (+ 204us) server_negotiation.cc:1035] Waiting for connection context
0620 14:47:17.399930 (+ 66us) server_negotiation.cc:299] Negotiation successful
0620 14:47:17.400109 (+ 179us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":4949,"thread_start_us":4798,"threads_started":1}
I20260620 14:47:17.404080 1089 tablet_service.cc:1467] Processing CreateTablet for tablet d956e7db98954bc48e6dbc253cad2b53 (DEFAULT_TABLE table=test-table [id=4402576131324f50951f3f5a5ec94d3a]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 14:47:17.404095 1356 tablet_service.cc:1467] Processing CreateTablet for tablet d956e7db98954bc48e6dbc253cad2b53 (DEFAULT_TABLE table=test-table [id=4402576131324f50951f3f5a5ec94d3a]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 14:47:17.404095 1223 tablet_service.cc:1467] Processing CreateTablet for tablet d956e7db98954bc48e6dbc253cad2b53 (DEFAULT_TABLE table=test-table [id=4402576131324f50951f3f5a5ec94d3a]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 14:47:17.405411 1089 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d956e7db98954bc48e6dbc253cad2b53. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 14:47:17.405447 1356 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d956e7db98954bc48e6dbc253cad2b53. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 14:47:17.405452 1223 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d956e7db98954bc48e6dbc253cad2b53. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 14:47:17.417968 1439 tablet_bootstrap.cc:492] T d956e7db98954bc48e6dbc253cad2b53 P 8fda64b6e2f44e8a9fa1e54a1575f881: Bootstrap starting.
I20260620 14:47:17.422577 1439 tablet_bootstrap.cc:654] T d956e7db98954bc48e6dbc253cad2b53 P 8fda64b6e2f44e8a9fa1e54a1575f881: Neither blocks nor log segments found. Creating new log.
I20260620 14:47:17.423779 1439 log.cc:824] T d956e7db98954bc48e6dbc253cad2b53 P 8fda64b6e2f44e8a9fa1e54a1575f881: Log is configured to *not* fsync() on all Append() calls
I20260620 14:47:17.426997 1439 tablet_bootstrap.cc:492] T d956e7db98954bc48e6dbc253cad2b53 P 8fda64b6e2f44e8a9fa1e54a1575f881: No bootstrap required, opened a new log
I20260620 14:47:17.427289 1439 ts_tablet_manager.cc:1397] T d956e7db98954bc48e6dbc253cad2b53 P 8fda64b6e2f44e8a9fa1e54a1575f881: Time spent bootstrapping tablet: real 0.010s user 0.003s sys 0.003s
I20260620 14:47:17.427632 1441 tablet_bootstrap.cc:492] T d956e7db98954bc48e6dbc253cad2b53 P d34fa4445c8e45918b1ecd8007e551a8: Bootstrap starting.
I20260620 14:47:17.431761 1441 tablet_bootstrap.cc:654] T d956e7db98954bc48e6dbc253cad2b53 P d34fa4445c8e45918b1ecd8007e551a8: Neither blocks nor log segments found. Creating new log.
I20260620 14:47:17.432883 1441 log.cc:824] T d956e7db98954bc48e6dbc253cad2b53 P d34fa4445c8e45918b1ecd8007e551a8: Log is configured to *not* fsync() on all Append() calls
I20260620 14:47:17.425295 1440 tablet_bootstrap.cc:492] T d956e7db98954bc48e6dbc253cad2b53 P e122103b272f4a58809f540448f227ff: Bootstrap starting.
I20260620 14:47:17.435897 1441 tablet_bootstrap.cc:492] T d956e7db98954bc48e6dbc253cad2b53 P d34fa4445c8e45918b1ecd8007e551a8: No bootstrap required, opened a new log
I20260620 14:47:17.436172 1441 ts_tablet_manager.cc:1397] T d956e7db98954bc48e6dbc253cad2b53 P d34fa4445c8e45918b1ecd8007e551a8: Time spent bootstrapping tablet: real 0.009s user 0.008s sys 0.000s
I20260620 14:47:17.438477 1439 raft_consensus.cc:348] T d956e7db98954bc48e6dbc253cad2b53 P 8fda64b6e2f44e8a9fa1e54a1575f881 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d34fa4445c8e45918b1ecd8007e551a8" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 35645 } } peers { permanent_uuid: "8fda64b6e2f44e8a9fa1e54a1575f881" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 46737 } } peers { permanent_uuid: "e122103b272f4a58809f540448f227ff" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 46475 } }
I20260620 14:47:17.439356 1440 tablet_bootstrap.cc:654] T d956e7db98954bc48e6dbc253cad2b53 P e122103b272f4a58809f540448f227ff: Neither blocks nor log segments found. Creating new log.
I20260620 14:47:17.439495 1439 raft_consensus.cc:374] T d956e7db98954bc48e6dbc253cad2b53 P 8fda64b6e2f44e8a9fa1e54a1575f881 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 14:47:17.439632 1439 raft_consensus.cc:729] T d956e7db98954bc48e6dbc253cad2b53 P 8fda64b6e2f44e8a9fa1e54a1575f881 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8fda64b6e2f44e8a9fa1e54a1575f881, State: Initialized, Role: FOLLOWER
I20260620 14:47:17.440351 1439 consensus_queue.cc:260] T d956e7db98954bc48e6dbc253cad2b53 P 8fda64b6e2f44e8a9fa1e54a1575f881 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d34fa4445c8e45918b1ecd8007e551a8" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 35645 } } peers { permanent_uuid: "8fda64b6e2f44e8a9fa1e54a1575f881" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 46737 } } peers { permanent_uuid: "e122103b272f4a58809f540448f227ff" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 46475 } }
I20260620 14:47:17.440740 1440 log.cc:824] T d956e7db98954bc48e6dbc253cad2b53 P e122103b272f4a58809f540448f227ff: Log is configured to *not* fsync() on all Append() calls
I20260620 14:47:17.443444 1439 ts_tablet_manager.cc:1428] T d956e7db98954bc48e6dbc253cad2b53 P 8fda64b6e2f44e8a9fa1e54a1575f881: Time spent starting tablet: real 0.016s user 0.007s sys 0.007s
I20260620 14:47:17.445443 1440 tablet_bootstrap.cc:492] T d956e7db98954bc48e6dbc253cad2b53 P e122103b272f4a58809f540448f227ff: No bootstrap required, opened a new log
I20260620 14:47:17.445416 1441 raft_consensus.cc:348] T d956e7db98954bc48e6dbc253cad2b53 P d34fa4445c8e45918b1ecd8007e551a8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d34fa4445c8e45918b1ecd8007e551a8" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 35645 } } peers { permanent_uuid: "8fda64b6e2f44e8a9fa1e54a1575f881" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 46737 } } peers { permanent_uuid: "e122103b272f4a58809f540448f227ff" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 46475 } }
I20260620 14:47:17.445757 1440 ts_tablet_manager.cc:1397] T d956e7db98954bc48e6dbc253cad2b53 P e122103b272f4a58809f540448f227ff: Time spent bootstrapping tablet: real 0.021s user 0.003s sys 0.004s
I20260620 14:47:17.445889 1441 raft_consensus.cc:374] T d956e7db98954bc48e6dbc253cad2b53 P d34fa4445c8e45918b1ecd8007e551a8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 14:47:17.446013 1441 raft_consensus.cc:729] T d956e7db98954bc48e6dbc253cad2b53 P d34fa4445c8e45918b1ecd8007e551a8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d34fa4445c8e45918b1ecd8007e551a8, State: Initialized, Role: FOLLOWER
I20260620 14:47:17.446712 1441 consensus_queue.cc:260] T d956e7db98954bc48e6dbc253cad2b53 P d34fa4445c8e45918b1ecd8007e551a8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d34fa4445c8e45918b1ecd8007e551a8" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 35645 } } peers { permanent_uuid: "8fda64b6e2f44e8a9fa1e54a1575f881" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 46737 } } peers { permanent_uuid: "e122103b272f4a58809f540448f227ff" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 46475 } }
I20260620 14:47:17.448135 1421 heartbeater.cc:503] Master 127.0.97.62:40505 was elected leader, sending a full tablet report...
I20260620 14:47:17.448563 1446 raft_consensus.cc:482] T d956e7db98954bc48e6dbc253cad2b53 P d34fa4445c8e45918b1ecd8007e551a8 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260620 14:47:17.448866 1446 raft_consensus.cc:504] T d956e7db98954bc48e6dbc253cad2b53 P d34fa4445c8e45918b1ecd8007e551a8 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d34fa4445c8e45918b1ecd8007e551a8" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 35645 } } peers { permanent_uuid: "8fda64b6e2f44e8a9fa1e54a1575f881" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 46737 } } peers { permanent_uuid: "e122103b272f4a58809f540448f227ff" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 46475 } }
I20260620 14:47:17.450954 1446 leader_election.cc:288] T d956e7db98954bc48e6dbc253cad2b53 P d34fa4445c8e45918b1ecd8007e551a8 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 8fda64b6e2f44e8a9fa1e54a1575f881 (127.0.97.1:46737), e122103b272f4a58809f540448f227ff (127.0.97.2:46475)
I20260620 14:47:17.454566 1441 ts_tablet_manager.cc:1428] T d956e7db98954bc48e6dbc253cad2b53 P d34fa4445c8e45918b1ecd8007e551a8: Time spent starting tablet: real 0.018s user 0.012s sys 0.000s
I20260620 14:47:17.457279 1440 raft_consensus.cc:348] T d956e7db98954bc48e6dbc253cad2b53 P e122103b272f4a58809f540448f227ff [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d34fa4445c8e45918b1ecd8007e551a8" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 35645 } } peers { permanent_uuid: "8fda64b6e2f44e8a9fa1e54a1575f881" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 46737 } } peers { permanent_uuid: "e122103b272f4a58809f540448f227ff" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 46475 } }
I20260620 14:47:17.457777 1440 raft_consensus.cc:374] T d956e7db98954bc48e6dbc253cad2b53 P e122103b272f4a58809f540448f227ff [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 14:47:17.457878 1440 raft_consensus.cc:729] T d956e7db98954bc48e6dbc253cad2b53 P e122103b272f4a58809f540448f227ff [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e122103b272f4a58809f540448f227ff, State: Initialized, Role: FOLLOWER
I20260620 14:47:17.458647 1440 consensus_queue.cc:260] T d956e7db98954bc48e6dbc253cad2b53 P e122103b272f4a58809f540448f227ff [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d34fa4445c8e45918b1ecd8007e551a8" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 35645 } } peers { permanent_uuid: "8fda64b6e2f44e8a9fa1e54a1575f881" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 46737 } } peers { permanent_uuid: "e122103b272f4a58809f540448f227ff" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 46475 } }
I20260620 14:47:17.459715 1423 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:47:17.451139 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:46737 (local address 127.0.97.3:57913)
0620 14:47:17.451313 (+ 174us) negotiation.cc:107] Waiting for socket to connect
0620 14:47:17.451334 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0620 14:47:17.451447 (+ 113us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 14:47:17.452597 (+ 1150us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 14:47:17.452602 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 14:47:17.452642 (+ 40us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 14:47:17.452836 (+ 194us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:47:17.452848 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:47:17.454262 (+ 1414us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:47:17.454267 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:47:17.456233 (+ 1966us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:47:17.456246 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:47:17.458908 (+ 2662us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:47:17.458914 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:47:17.459115 (+ 201us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:47:17.459153 (+ 38us) client_negotiation.cc:769] Sending connection context
0620 14:47:17.459271 (+ 118us) client_negotiation.cc:240] Negotiation successful
0620 14:47:17.459415 (+ 144us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":43}
I20260620 14:47:17.459717 1436 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:47:17.451250 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.3:57913 (local address 127.0.97.1:46737)
0620 14:47:17.452003 (+ 753us) server_negotiation.cc:206] Beginning negotiation
0620 14:47:17.452011 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0620 14:47:17.452034 (+ 23us) server_negotiation.cc:407] Connection header received
0620 14:47:17.452110 (+ 76us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:47:17.452118 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:47:17.452200 (+ 82us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:47:17.452296 (+ 96us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 14:47:17.452962 (+ 666us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:47:17.453995 (+ 1033us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:47:17.456393 (+ 2398us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:47:17.457988 (+ 1595us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:47:17.458121 (+ 133us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:47:17.458257 (+ 136us) server_negotiation.cc:1035] Waiting for connection context
0620 14:47:17.459309 (+ 1052us) server_negotiation.cc:299] Negotiation successful
0620 14:47:17.459442 (+ 133us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":587}
I20260620 14:47:17.460974 1109 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d956e7db98954bc48e6dbc253cad2b53" candidate_uuid: "d34fa4445c8e45918b1ecd8007e551a8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8fda64b6e2f44e8a9fa1e54a1575f881" is_pre_election: true
I20260620 14:47:17.461598 1109 raft_consensus.cc:2454] T d956e7db98954bc48e6dbc253cad2b53 P 8fda64b6e2f44e8a9fa1e54a1575f881 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d34fa4445c8e45918b1ecd8007e551a8 in term 0.
I20260620 14:47:17.462071 1447 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:47:17.451652 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.2:46475 (local address 127.0.97.3:53505)
0620 14:47:17.452354 (+ 702us) negotiation.cc:107] Waiting for socket to connect
0620 14:47:17.452373 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0620 14:47:17.452508 (+ 135us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 14:47:17.453834 (+ 1326us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 14:47:17.453840 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 14:47:17.453865 (+ 25us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 14:47:17.455001 (+ 1136us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:47:17.455013 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:47:17.456574 (+ 1561us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:47:17.456579 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:47:17.458919 (+ 2340us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:47:17.458934 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:47:17.461230 (+ 2296us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:47:17.461238 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:47:17.461430 (+ 192us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:47:17.461465 (+ 35us) client_negotiation.cc:769] Sending connection context
0620 14:47:17.461560 (+ 95us) client_negotiation.cc:240] Negotiation successful
0620 14:47:17.461714 (+ 154us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":548,"thread_start_us":1578,"threads_started":1}
I20260620 14:47:17.462229 1437 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:47:17.453178 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.3:53505 (local address 127.0.97.2:46475)
0620 14:47:17.453403 (+ 225us) server_negotiation.cc:206] Beginning negotiation
0620 14:47:17.453409 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 14:47:17.453430 (+ 21us) server_negotiation.cc:407] Connection header received
0620 14:47:17.453497 (+ 67us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:47:17.453502 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:47:17.453578 (+ 76us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:47:17.453679 (+ 101us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 14:47:17.455212 (+ 1533us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:47:17.456375 (+ 1163us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:47:17.459104 (+ 2729us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:47:17.460791 (+ 1687us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:47:17.460921 (+ 130us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:47:17.461026 (+ 105us) server_negotiation.cc:1035] Waiting for connection context
0620 14:47:17.461649 (+ 623us) server_negotiation.cc:299] Negotiation successful
0620 14:47:17.461854 (+ 205us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":73}
I20260620 14:47:17.462472 1310 leader_election.cc:302] T d956e7db98954bc48e6dbc253cad2b53 P d34fa4445c8e45918b1ecd8007e551a8 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 8fda64b6e2f44e8a9fa1e54a1575f881, d34fa4445c8e45918b1ecd8007e551a8; no voters:
I20260620 14:47:17.462926 1446 raft_consensus.cc:2781] T d956e7db98954bc48e6dbc253cad2b53 P d34fa4445c8e45918b1ecd8007e551a8 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260620 14:47:17.463132 1446 raft_consensus.cc:482] T d956e7db98954bc48e6dbc253cad2b53 P d34fa4445c8e45918b1ecd8007e551a8 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260620 14:47:17.463272 1446 raft_consensus.cc:3037] T d956e7db98954bc48e6dbc253cad2b53 P d34fa4445c8e45918b1ecd8007e551a8 [term 0 FOLLOWER]: Advancing to term 1
I20260620 14:47:17.463819 1288 heartbeater.cc:503] Master 127.0.97.62:40505 was elected leader, sending a full tablet report...
I20260620 14:47:17.463814 1243 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d956e7db98954bc48e6dbc253cad2b53" candidate_uuid: "d34fa4445c8e45918b1ecd8007e551a8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e122103b272f4a58809f540448f227ff" is_pre_election: true
I20260620 14:47:17.464418 1243 raft_consensus.cc:2454] T d956e7db98954bc48e6dbc253cad2b53 P e122103b272f4a58809f540448f227ff [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d34fa4445c8e45918b1ecd8007e551a8 in term 0.
I20260620 14:47:17.464666 1440 ts_tablet_manager.cc:1428] T d956e7db98954bc48e6dbc253cad2b53 P e122103b272f4a58809f540448f227ff: Time spent starting tablet: real 0.019s user 0.018s sys 0.001s
I20260620 14:47:17.466698 1446 raft_consensus.cc:504] T d956e7db98954bc48e6dbc253cad2b53 P d34fa4445c8e45918b1ecd8007e551a8 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d34fa4445c8e45918b1ecd8007e551a8" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 35645 } } peers { permanent_uuid: "8fda64b6e2f44e8a9fa1e54a1575f881" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 46737 } } peers { permanent_uuid: "e122103b272f4a58809f540448f227ff" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 46475 } }
I20260620 14:47:17.467381 1446 leader_election.cc:288] T d956e7db98954bc48e6dbc253cad2b53 P d34fa4445c8e45918b1ecd8007e551a8 [CANDIDATE]: Term 1 election: Requested vote from peers 8fda64b6e2f44e8a9fa1e54a1575f881 (127.0.97.1:46737), e122103b272f4a58809f540448f227ff (127.0.97.2:46475)
I20260620 14:47:17.467989 1243 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d956e7db98954bc48e6dbc253cad2b53" candidate_uuid: "d34fa4445c8e45918b1ecd8007e551a8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e122103b272f4a58809f540448f227ff"
I20260620 14:47:17.468066 1109 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d956e7db98954bc48e6dbc253cad2b53" candidate_uuid: "d34fa4445c8e45918b1ecd8007e551a8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8fda64b6e2f44e8a9fa1e54a1575f881"
I20260620 14:47:17.468204 1243 raft_consensus.cc:3037] T d956e7db98954bc48e6dbc253cad2b53 P e122103b272f4a58809f540448f227ff [term 0 FOLLOWER]: Advancing to term 1
I20260620 14:47:17.468348 1109 raft_consensus.cc:3037] T d956e7db98954bc48e6dbc253cad2b53 P 8fda64b6e2f44e8a9fa1e54a1575f881 [term 0 FOLLOWER]: Advancing to term 1
I20260620 14:47:17.471453 1243 raft_consensus.cc:2454] T d956e7db98954bc48e6dbc253cad2b53 P e122103b272f4a58809f540448f227ff [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d34fa4445c8e45918b1ecd8007e551a8 in term 1.
I20260620 14:47:17.471575 1109 raft_consensus.cc:2454] T d956e7db98954bc48e6dbc253cad2b53 P 8fda64b6e2f44e8a9fa1e54a1575f881 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d34fa4445c8e45918b1ecd8007e551a8 in term 1.
I20260620 14:47:17.471940 1310 leader_election.cc:302] T d956e7db98954bc48e6dbc253cad2b53 P d34fa4445c8e45918b1ecd8007e551a8 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: d34fa4445c8e45918b1ecd8007e551a8, e122103b272f4a58809f540448f227ff; no voters:
I20260620 14:47:17.472270 1446 raft_consensus.cc:2781] T d956e7db98954bc48e6dbc253cad2b53 P d34fa4445c8e45918b1ecd8007e551a8 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 14:47:17.472698 1446 raft_consensus.cc:686] T d956e7db98954bc48e6dbc253cad2b53 P d34fa4445c8e45918b1ecd8007e551a8 [term 1 LEADER]: Becoming Leader. State: Replica: d34fa4445c8e45918b1ecd8007e551a8, State: Running, Role: LEADER
I20260620 14:47:17.473248 1446 consensus_queue.cc:237] T d956e7db98954bc48e6dbc253cad2b53 P d34fa4445c8e45918b1ecd8007e551a8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d34fa4445c8e45918b1ecd8007e551a8" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 35645 } } peers { permanent_uuid: "8fda64b6e2f44e8a9fa1e54a1575f881" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 46737 } } peers { permanent_uuid: "e122103b272f4a58809f540448f227ff" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 46475 } }
I20260620 14:47:17.477792 964 catalog_manager.cc:5495] T d956e7db98954bc48e6dbc253cad2b53 P d34fa4445c8e45918b1ecd8007e551a8 reported cstate change: term changed from 0 to 1, leader changed from <none> to d34fa4445c8e45918b1ecd8007e551a8 (127.0.97.3). New cstate: current_term: 1 leader_uuid: "d34fa4445c8e45918b1ecd8007e551a8" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d34fa4445c8e45918b1ecd8007e551a8" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 35645 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "8fda64b6e2f44e8a9fa1e54a1575f881" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 46737 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "e122103b272f4a58809f540448f227ff" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 46475 } health_report { overall_health: UNKNOWN } } }
Jun 20 14:47:17 dist-test-slave-q9j6 krb5kdc[920](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781966835, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.1@KRBTEST.COM
I20260620 14:47:17.508913 1436 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:47:17.493477 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37564 (local address 127.0.97.1:46737)
0620 14:47:17.493644 (+ 167us) server_negotiation.cc:206] Beginning negotiation
0620 14:47:17.493648 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 14:47:17.494138 (+ 490us) server_negotiation.cc:407] Connection header received
0620 14:47:17.494266 (+ 128us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:47:17.494270 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:47:17.494350 (+ 80us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:47:17.494438 (+ 88us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:47:17.495922 (+ 1484us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:47:17.496676 (+ 754us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:47:17.498475 (+ 1799us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:47:17.498900 (+ 425us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:47:17.499003 (+ 103us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:47:17.503029 (+ 4026us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 14:47:17.503045 (+ 16us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 14:47:17.503062 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 14:47:17.503111 (+ 49us) server_negotiation.cc:922] calling sasl_server_start()
0620 14:47:17.506216 (+ 3105us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:47:17.506893 (+ 677us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:47:17.506901 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:47:17.506903 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:47:17.507049 (+ 146us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:47:17.507386 (+ 337us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:47:17.507410 (+ 24us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:47:17.507412 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:47:17.507883 (+ 471us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 14:47:17.508133 (+ 250us) server_negotiation.cc:1035] Waiting for connection context
0620 14:47:17.508360 (+ 227us) server_negotiation.cc:299] Negotiation successful
0620 14:47:17.508530 (+ 170us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":63}
Jun 20 14:47:17 dist-test-slave-q9j6 krb5kdc[920](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781966837, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-user@KRBTEST.COM:
W20260620 14:47:17.544328 1155 tablet.cc:2369] T d956e7db98954bc48e6dbc253cad2b53 P 8fda64b6e2f44e8a9fa1e54a1575f881: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260620 14:47:17.547900 1289 tablet.cc:2369] T d956e7db98954bc48e6dbc253cad2b53 P e122103b272f4a58809f540448f227ff: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260620 14:47:17.549592 1422 tablet.cc:2369] T d956e7db98954bc48e6dbc253cad2b53 P d34fa4445c8e45918b1ecd8007e551a8: Can't schedule compaction. Clean time has not been advanced past its initial value.
Jun 20 14:47:17 dist-test-slave-q9j6 krb5kdc[920](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781966837, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.0.97.1@KRBTEST.COM
I20260620 14:47:17.557906 1436 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:47:17.543042 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37580 (local address 127.0.97.1:46737)
0620 14:47:17.543269 (+ 227us) server_negotiation.cc:206] Beginning negotiation
0620 14:47:17.543278 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0620 14:47:17.543529 (+ 251us) server_negotiation.cc:407] Connection header received
0620 14:47:17.543808 (+ 279us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:47:17.543812 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:47:17.543875 (+ 63us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:47:17.543985 (+ 110us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:47:17.545430 (+ 1445us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:47:17.546169 (+ 739us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:47:17.547955 (+ 1786us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:47:17.548447 (+ 492us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:47:17.548594 (+ 147us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:47:17.552490 (+ 3896us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 14:47:17.552496 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 14:47:17.552499 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 14:47:17.552537 (+ 38us) server_negotiation.cc:922] calling sasl_server_start()
0620 14:47:17.555162 (+ 2625us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:47:17.555918 (+ 756us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:47:17.555923 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:47:17.555925 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:47:17.556062 (+ 137us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:47:17.556492 (+ 430us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:47:17.556496 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:47:17.556498 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:47:17.556766 (+ 268us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 14:47:17.557039 (+ 273us) server_negotiation.cc:1035] Waiting for connection context
0620 14:47:17.557333 (+ 294us) server_negotiation.cc:299] Negotiation successful
0620 14:47:17.557558 (+ 225us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":74}
Jun 20 14:47:17 dist-test-slave-q9j6 krb5kdc[920](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781966837, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
Jun 20 14:47:17 dist-test-slave-q9j6 krb5kdc[920](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781966837, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.1@KRBTEST.COM
I20260620 14:47:17.603120 1436 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:47:17.588297 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37596 (local address 127.0.97.1:46737)
0620 14:47:17.588538 (+ 241us) server_negotiation.cc:206] Beginning negotiation
0620 14:47:17.588547 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0620 14:47:17.588860 (+ 313us) server_negotiation.cc:407] Connection header received
0620 14:47:17.589075 (+ 215us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:47:17.589079 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:47:17.589147 (+ 68us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:47:17.589264 (+ 117us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:47:17.590687 (+ 1423us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:47:17.591379 (+ 692us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:47:17.592739 (+ 1360us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:47:17.593170 (+ 431us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:47:17.593311 (+ 141us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:47:17.597281 (+ 3970us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 14:47:17.597287 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 14:47:17.597290 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 14:47:17.597328 (+ 38us) server_negotiation.cc:922] calling sasl_server_start()
0620 14:47:17.600451 (+ 3123us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:47:17.601168 (+ 717us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:47:17.601173 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:47:17.601175 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:47:17.601301 (+ 126us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:47:17.601665 (+ 364us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:47:17.601673 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:47:17.601677 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:47:17.601964 (+ 287us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 14:47:17.602254 (+ 290us) server_negotiation.cc:1035] Waiting for connection context
0620 14:47:17.602533 (+ 279us) server_negotiation.cc:299] Negotiation successful
0620 14:47:17.602775 (+ 242us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":59}
W20260620 14:47:17.604897 1069 tablet_service.cc:2908] Rejecting scan request for tablet d956e7db98954bc48e6dbc253cad2b53: Uninitialized: clean time has not yet been initialized
I20260620 14:47:17.607801 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskUci3DU/build/asan/bin/kudu pid 1026
I20260620 14:47:17.620893 1436 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:47:17.608731 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37600 (local address 127.0.97.1:46737)
0620 14:47:17.608901 (+ 170us) server_negotiation.cc:206] Beginning negotiation
0620 14:47:17.608907 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 14:47:17.609289 (+ 382us) server_negotiation.cc:407] Connection header received
0620 14:47:17.609491 (+ 202us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:47:17.609496 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:47:17.609575 (+ 79us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:47:17.609679 (+ 104us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:47:17.610842 (+ 1163us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:47:17.611542 (+ 700us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:47:17.612869 (+ 1327us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:47:17.613309 (+ 440us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:47:17.613459 (+ 150us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:47:17.615703 (+ 2244us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 14:47:17.615708 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 14:47:17.615711 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 14:47:17.615755 (+ 44us) server_negotiation.cc:922] calling sasl_server_start()
0620 14:47:17.618081 (+ 2326us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:47:17.618805 (+ 724us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:47:17.618810 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:47:17.618814 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:47:17.618947 (+ 133us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:47:17.619324 (+ 377us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:47:17.619329 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:47:17.619336 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:47:17.619735 (+ 399us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 14:47:17.620023 (+ 288us) server_negotiation.cc:1035] Waiting for connection context
0620 14:47:17.620318 (+ 295us) server_negotiation.cc:299] Negotiation successful
0620 14:47:17.620556 (+ 238us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":59}
I20260620 14:47:17.621537 1149 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 14:47:17.711781 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskUci3DU/build/asan/bin/kudu with pid 1026
I20260620 14:47:17.728778 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskUci3DU/build/asan/bin/kudu pid 1160
Jun 20 14:47:17 dist-test-slave-q9j6 krb5kdc[920](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781966837, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.2@KRBTEST.COM
I20260620 14:47:17.744508 1437 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:47:17.729630 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37660 (local address 127.0.97.2:46475)
0620 14:47:17.729843 (+ 213us) server_negotiation.cc:206] Beginning negotiation
0620 14:47:17.729851 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0620 14:47:17.729877 (+ 26us) server_negotiation.cc:407] Connection header received
0620 14:47:17.730109 (+ 232us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:47:17.730113 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:47:17.730176 (+ 63us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:47:17.730286 (+ 110us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:47:17.731420 (+ 1134us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:47:17.732448 (+ 1028us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:47:17.733708 (+ 1260us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:47:17.734105 (+ 397us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:47:17.734240 (+ 135us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:47:17.738285 (+ 4045us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 14:47:17.738298 (+ 13us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 14:47:17.738315 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 14:47:17.738359 (+ 44us) server_negotiation.cc:922] calling sasl_server_start()
0620 14:47:17.741349 (+ 2990us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:47:17.742161 (+ 812us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:47:17.742177 (+ 16us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:47:17.742180 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:47:17.742319 (+ 139us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:47:17.742738 (+ 419us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:47:17.742745 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:47:17.742748 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:47:17.743228 (+ 480us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 14:47:17.743559 (+ 331us) server_negotiation.cc:1035] Waiting for connection context
0620 14:47:17.743930 (+ 371us) server_negotiation.cc:299] Negotiation successful
0620 14:47:17.744148 (+ 218us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":94}
I20260620 14:47:17.745151 1283 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 14:47:17.831928 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskUci3DU/build/asan/bin/kudu with pid 1160
I20260620 14:47:17.848568 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskUci3DU/build/asan/bin/kudu pid 1293
Jun 20 14:47:17 dist-test-slave-q9j6 krb5kdc[920](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781966837, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.3@KRBTEST.COM
I20260620 14:47:17.863564 1435 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:47:17.849430 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38482 (local address 127.0.97.3:35645)
0620 14:47:17.849578 (+ 148us) server_negotiation.cc:206] Beginning negotiation
0620 14:47:17.849584 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 14:47:17.849604 (+ 20us) server_negotiation.cc:407] Connection header received
0620 14:47:17.849728 (+ 124us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:47:17.849733 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:47:17.849804 (+ 71us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:47:17.849908 (+ 104us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:47:17.851106 (+ 1198us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:47:17.851904 (+ 798us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:47:17.853171 (+ 1267us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:47:17.853580 (+ 409us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:47:17.853721 (+ 141us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:47:17.858057 (+ 4336us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 14:47:17.858074 (+ 17us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 14:47:17.858090 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 14:47:17.858128 (+ 38us) server_negotiation.cc:922] calling sasl_server_start()
0620 14:47:17.860828 (+ 2700us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:47:17.861517 (+ 689us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:47:17.861525 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:47:17.861528 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:47:17.861659 (+ 131us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:47:17.862062 (+ 403us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:47:17.862068 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:47:17.862071 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:47:17.862441 (+ 370us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 14:47:17.862738 (+ 297us) server_negotiation.cc:1035] Waiting for connection context
0620 14:47:17.863013 (+ 275us) server_negotiation.cc:299] Negotiation successful
0620 14:47:17.863255 (+ 242us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":37}
I20260620 14:47:17.864327 1416 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 14:47:17.953428 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskUci3DU/build/asan/bin/kudu with pid 1293
I20260620 14:47:17.969640 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskUci3DU/build/asan/bin/kudu pid 933
I20260620 14:47:17.970769 995 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 14:47:18.077324 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskUci3DU/build/asan/bin/kudu with pid 933
2026-06-20T14:47:18Z chronyd exiting
[ OK ] SecurityITest.TestAuthorizationOnChecksum (4520 ms)
[ RUN ] SecurityITest.SmokeTestAsAuthorizedUser
Loading random data
Initializing database '/tmp/dist-test-taskUci3DU/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781966829091300-388-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 20 14:47:18 dist-test-slave-q9j6 krb5kdc[1468](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 20 14:47:18 dist-test-slave-q9j6 krb5kdc[1468](info): set up 2 sockets
Jun 20 14:47:18 dist-test-slave-q9j6 krb5kdc[1468](info): commencing operation
krb5kdc: starting...
W20260620 14:47:19.171569 388 mini_kdc.cc:121] Time spent starting KDC: real 1.051s user 0.002s sys 0.013s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskUci3DU/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781966829091300-388-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskUci3DU/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781966829091300-388-0/krb5kdc/test-user.keytab.
Jun 20 14:47:19 dist-test-slave-q9j6 krb5kdc[1468](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781966839, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-20T14:47:19Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T14:47:19Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.97.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskUci3DU/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781966829091300-388-0/krb5kdc/kudu_127.0.97.62.keytab.
Entry for principal kudu/127.0.97.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskUci3DU/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781966829091300-388-0/krb5kdc/kudu_127.0.97.62.keytab.
I20260620 14:47:19.351022 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskUci3DU/build/asan/bin/kudu
/tmp/dist-test-taskUci3DU/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskUci3DU/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781966829091300-388-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskUci3DU/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781966829091300-388-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskUci3DU/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781966829091300-388-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskUci3DU/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781966829091300-388-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:37781
--webserver_interface=127.0.97.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskUci3DU/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781966829091300-388-0/krb5kdc/kudu_127.0.97.62.keytab
--principal=kudu/127.0.97.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:37215
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.97.62:37781
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--txn_manager_enabled=true with env {KRB5CCNAME=/tmp/dist-test-taskUci3DU/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781966829091300-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskUci3DU/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781966829091300-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskUci3DU/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781966829091300-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 14:47:19.674355 1481 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 14:47:19.674691 1481 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 14:47:19.674873 1481 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 14:47:19.682405 1481 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 14:47:19.682526 1481 flags.cc:425] Enabled experimental flag: --txn_manager_enabled=true
W20260620 14:47:19.682612 1481 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 14:47:19.682718 1481 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 14:47:19.682804 1481 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 14:47:19.682919 1481 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 14:47:19.693058 1481 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:37215
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskUci3DU/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781966829091300-388-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskUci3DU/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781966829091300-388-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.97.62:37781
--txn_manager_enabled=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskUci3DU/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781966829091300-388-0/krb5kdc/kudu_127.0.97.62.keytab
--principal=kudu/127.0.97.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:37781
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskUci3DU/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781966829091300-388-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.97.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskUci3DU/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781966829091300-388-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 13:41:20 UTC on ccycloud.quasar-vdagoi.root.comops.site
build id 4457462
ASAN enabled
I20260620 14:47:19.694474 1481 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 14:47:19.695904 1481 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 14:47:19.708251 1486 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 14:47:19.708212 1487 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 14:47:19.708209 1489 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 14:47:19.709241 1481 server_base.cc:1034] running on GCE node
I20260620 14:47:19.710119 1481 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 14:47:19.711651 1481 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 14:47:19.712918 1481 hybrid_clock.cc:648] HybridClock initialized: now 1781966839712839 us; error 65 us; skew 500 ppm
I20260620 14:47:19.713279 1481 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 14:47:19 dist-test-slave-q9j6 krb5kdc[1468](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781966839, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 14:47:19.716495 1481 init.cc:377] Logged in from keytab as kudu/127.0.97.62@KRBTEST.COM (short username kudu)
I20260620 14:47:19.718063 1481 webserver.cc:460] Webserver started at http://127.0.97.62:44955/ using document root <none> and password file <none>
I20260620 14:47:19.718833 1481 fs_manager.cc:359] Metadata directory not provided
I20260620 14:47:19.718964 1481 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 14:47:19.719336 1481 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 14:47:19.722431 1481 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskUci3DU/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781966829091300-388-0/minicluster-data/master-0/data/instance:
uuid: "8f6b9a65e0374b0b9f1f530836785c11"
format_stamp: "Formatted at 2026-06-20 14:47:19 on dist-test-slave-q9j6"
server_key: "b303f93165b7fca8cae753798ec244a5"
server_key_iv: "21d6e6df42bbdfd68a88d89f3952cffc"
server_key_version: "encryptionkey@0"
I20260620 14:47:19.723251 1481 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskUci3DU/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781966829091300-388-0/minicluster-data/master-0/wal/instance:
uuid: "8f6b9a65e0374b0b9f1f530836785c11"
format_stamp: "Formatted at 2026-06-20 14:47:19 on dist-test-slave-q9j6"
server_key: "b303f93165b7fca8cae753798ec244a5"
server_key_iv: "21d6e6df42bbdfd68a88d89f3952cffc"
server_key_version: "encryptionkey@0"
I20260620 14:47:19.728544 1481 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.003s sys 0.001s
I20260620 14:47:19.732043 1496 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 14:47:19.733378 1481 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.002s
I20260620 14:47:19.733568 1481 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskUci3DU/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781966829091300-388-0/minicluster-data/master-0/data,/tmp/dist-test-taskUci3DU/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781966829091300-388-0/minicluster-data/master-0/wal
uuid: "8f6b9a65e0374b0b9f1f530836785c11"
format_stamp: "Formatted at 2026-06-20 14:47:19 on dist-test-slave-q9j6"
server_key: "b303f93165b7fca8cae753798ec244a5"
server_key_iv: "21d6e6df42bbdfd68a88d89f3952cffc"
server_key_version: "encryptionkey@0"
I20260620 14:47:19.733794 1481 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskUci3DU/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781966829091300-388-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskUci3DU/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781966829091300-388-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskUci3DU/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781966829091300-388-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 14:47:19.763458 1481 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 14:47:19.767796 1481 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 14:47:19.768083 1481 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 14:47:19.790287 1481 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.62:37781
I20260620 14:47:19.790266 1557 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.62:37781 every 8 connection(s)
I20260620 14:47:19.791752 1481 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskUci3DU/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781966829091300-388-0/minicluster-data/master-0/data/info.pb
I20260620 14:47:19.795562 1558 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 14:47:19.799939 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskUci3DU/build/asan/bin/kudu as pid 1481
I20260620 14:47:19.800158 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskUci3DU/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781966829091300-388-0/minicluster-data/master-0/wal/instance
I20260620 14:47:19.800825 388 external_mini_cluster.cc:1428] Setting key 9929d31b4f9dd682e0cd7953a4e86e8f
I20260620 14:47:19.807551 1558 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 8f6b9a65e0374b0b9f1f530836785c11: Bootstrap starting.
I20260620 14:47:19.811563 1558 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 8f6b9a65e0374b0b9f1f530836785c11: Neither blocks nor log segments found. Creating new log.
I20260620 14:47:19.812856 1558 log.cc:824] T 00000000000000000000000000000000 P 8f6b9a65e0374b0b9f1f530836785c11: Log is configured to *not* fsync() on all Append() calls
I20260620 14:47:19.816624 1558 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 8f6b9a65e0374b0b9f1f530836785c11: No bootstrap required, opened a new log
Jun 20 14:47:19 dist-test-slave-q9j6 krb5kdc[1468](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781966839, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260620 14:47:19.825874 1558 raft_consensus.cc:348] T 00000000000000000000000000000000 P 8f6b9a65e0374b0b9f1f530836785c11 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8f6b9a65e0374b0b9f1f530836785c11" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 37781 } }
I20260620 14:47:19.826267 1561 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:47:19.803337 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56562 (local address 127.0.97.62:37781)
0620 14:47:19.804279 (+ 942us) server_negotiation.cc:206] Beginning negotiation
0620 14:47:19.804292 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0620 14:47:19.804340 (+ 48us) server_negotiation.cc:407] Connection header received
0620 14:47:19.806162 (+ 1822us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:47:19.806188 (+ 26us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:47:19.806638 (+ 450us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:47:19.806994 (+ 356us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:47:19.808886 (+ 1892us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:47:19.810302 (+ 1416us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:47:19.812140 (+ 1838us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:47:19.812892 (+ 752us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:47:19.813144 (+ 252us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:47:19.818089 (+ 4945us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 14:47:19.818098 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 14:47:19.818116 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 14:47:19.818178 (+ 62us) server_negotiation.cc:922] calling sasl_server_start()
0620 14:47:19.821702 (+ 3524us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:47:19.822647 (+ 945us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:47:19.822653 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:47:19.822661 (+ 8us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:47:19.822853 (+ 192us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:47:19.823255 (+ 402us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:47:19.823258 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:47:19.823260 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:47:19.823967 (+ 707us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 14:47:19.824209 (+ 242us) server_negotiation.cc:1035] Waiting for connection context
0620 14:47:19.824674 (+ 465us) server_negotiation.cc:299] Negotiation successful
0620 14:47:19.825134 (+ 460us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":710,"thread_start_us":475,"threads_started":1}
I20260620 14:47:19.826298 1558 raft_consensus.cc:374] T 00000000000000000000000000000000 P 8f6b9a65e0374b0b9f1f530836785c11 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 14:47:19.826421 1558 raft_consensus.cc:729] T 00000000000000000000000000000000 P 8f6b9a65e0374b0b9f1f530836785c11 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8f6b9a65e0374b0b9f1f530836785c11, State: Initialized, Role: FOLLOWER
I20260620 14:47:19.827107 1558 consensus_queue.cc:260] T 00000000000000000000000000000000 P 8f6b9a65e0374b0b9f1f530836785c11 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8f6b9a65e0374b0b9f1f530836785c11" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 37781 } }
I20260620 14:47:19.827389 1558 raft_consensus.cc:388] T 00000000000000000000000000000000 P 8f6b9a65e0374b0b9f1f530836785c11 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 14:47:19.827555 1558 raft_consensus.cc:482] T 00000000000000000000000000000000 P 8f6b9a65e0374b0b9f1f530836785c11 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 14:47:19.827791 1558 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 8f6b9a65e0374b0b9f1f530836785c11 [term 0 FOLLOWER]: Advancing to term 1
I20260620 14:47:19.830117 1558 raft_consensus.cc:504] T 00000000000000000000000000000000 P 8f6b9a65e0374b0b9f1f530836785c11 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8f6b9a65e0374b0b9f1f530836785c11" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 37781 } }
I20260620 14:47:19.830643 1558 leader_election.cc:302] T 00000000000000000000000000000000 P 8f6b9a65e0374b0b9f1f530836785c11 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 8f6b9a65e0374b0b9f1f530836785c11; no voters:
I20260620 14:47:19.831362 1558 leader_election.cc:288] T 00000000000000000000000000000000 P 8f6b9a65e0374b0b9f1f530836785c11 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260620 14:47:19.831532 1563 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 8f6b9a65e0374b0b9f1f530836785c11 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 14:47:19.832326 1563 raft_consensus.cc:686] T 00000000000000000000000000000000 P 8f6b9a65e0374b0b9f1f530836785c11 [term 1 LEADER]: Becoming Leader. State: Replica: 8f6b9a65e0374b0b9f1f530836785c11, State: Running, Role: LEADER
I20260620 14:47:19.832928 1563 consensus_queue.cc:237] T 00000000000000000000000000000000 P 8f6b9a65e0374b0b9f1f530836785c11 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8f6b9a65e0374b0b9f1f530836785c11" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 37781 } }
I20260620 14:47:19.833711 1558 sys_catalog.cc:564] T 00000000000000000000000000000000 P 8f6b9a65e0374b0b9f1f530836785c11 [sys.catalog]: configured and running, proceeding with master startup.
I20260620 14:47:19.836130 1564 sys_catalog.cc:455] T 00000000000000000000000000000000 P 8f6b9a65e0374b0b9f1f530836785c11 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "8f6b9a65e0374b0b9f1f530836785c11" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8f6b9a65e0374b0b9f1f530836785c11" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 37781 } } }
I20260620 14:47:19.836620 1564 sys_catalog.cc:458] T 00000000000000000000000000000000 P 8f6b9a65e0374b0b9f1f530836785c11 [sys.catalog]: This master's current role is: LEADER
I20260620 14:47:19.836493 1565 sys_catalog.cc:455] T 00000000000000000000000000000000 P 8f6b9a65e0374b0b9f1f530836785c11 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 8f6b9a65e0374b0b9f1f530836785c11. Latest consensus state: current_term: 1 leader_uuid: "8f6b9a65e0374b0b9f1f530836785c11" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8f6b9a65e0374b0b9f1f530836785c11" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 37781 } } }
I20260620 14:47:19.836757 1565 sys_catalog.cc:458] T 00000000000000000000000000000000 P 8f6b9a65e0374b0b9f1f530836785c11 [sys.catalog]: This master's current role is: LEADER
I20260620 14:47:19.838639 1570 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 14:47:19.843786 1570 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 14:47:19.853778 1570 catalog_manager.cc:1347] Generated new cluster ID: f86479c63abc40af86436a40eeeb19f1
I20260620 14:47:19.853919 1570 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 14:47:19.862010 1570 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 14:47:19.863950 1570 catalog_manager.cc:1504] Loading token signing keys...
I20260620 14:47:19.875581 1570 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 8f6b9a65e0374b0b9f1f530836785c11: Generated new TSK 0
I20260620 14:47:19.876777 1570 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.0.97.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskUci3DU/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781966829091300-388-0/krb5kdc/kudu_127.0.97.1.keytab.
Entry for principal kudu/127.0.97.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskUci3DU/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781966829091300-388-0/krb5kdc/kudu_127.0.97.1.keytab.
I20260620 14:47:19.921552 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskUci3DU/build/asan/bin/kudu
/tmp/dist-test-taskUci3DU/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskUci3DU/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781966829091300-388-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskUci3DU/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781966829091300-388-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskUci3DU/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781966829091300-388-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskUci3DU/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781966829091300-388-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.1:0
--local_ip_for_outbound_sockets=127.0.97.1
--webserver_interface=127.0.97.1
--webserver_port=0
--tserver_master_addrs=127.0.97.62:37781
--keytab_file=/tmp/dist-test-taskUci3DU/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781966829091300-388-0/krb5kdc/kudu_127.0.97.1.keytab
--principal=kudu/127.0.97.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:37215
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-taskUci3DU/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781966829091300-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskUci3DU/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781966829091300-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskUci3DU/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781966829091300-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 14:47:20.260288 1584 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 14:47:20.260589 1584 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 14:47:20.260753 1584 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 14:47:20.268242 1584 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 14:47:20.268409 1584 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 14:47:20.268494 1584 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260620 14:47:20.268576 1584 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.1
I20260620 14:47:20.278542 1584 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:37215
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskUci3DU/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781966829091300-388-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskUci3DU/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781966829091300-388-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskUci3DU/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781966829091300-388-0/krb5kdc/kudu_127.0.97.1.keytab
--principal=kudu/127.0.97.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskUci3DU/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781966829091300-388-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.97.1
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.0.97.62:37781
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.1
--log_dir=/tmp/dist-test-taskUci3DU/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781966829091300-388-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 13:41:20 UTC on ccycloud.quasar-vdagoi.root.comops.site
build id 4457462
ASAN enabled
I20260620 14:47:20.280025 1584 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 14:47:20.281388 1584 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 14:47:20.295099 1592 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 14:47:20.295099 1590 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 14:47:20.295148 1589 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 14:47:20.296025 1584 server_base.cc:1034] running on GCE node
I20260620 14:47:20.296700 1584 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 14:47:20.297721 1584 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 14:47:20.298985 1584 hybrid_clock.cc:648] HybridClock initialized: now 1781966840298928 us; error 46 us; skew 500 ppm
I20260620 14:47:20.299324 1584 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 14:47:20 dist-test-slave-q9j6 krb5kdc[1468](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781966840, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 14:47:20.305868 1584 init.cc:377] Logged in from keytab as kudu/127.0.97.1@KRBTEST.COM (short username kudu)
I20260620 14:47:20.307387 1584 webserver.cc:460] Webserver started at http://127.0.97.1:34037/ using document root <none> and password file <none>
I20260620 14:47:20.308204 1584 fs_manager.cc:359] Metadata directory not provided
I20260620 14:47:20.308301 1584 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 14:47:20.308596 1584 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 14:47:20.311715 1584 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskUci3DU/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781966829091300-388-0/minicluster-data/ts-0/data/instance:
uuid: "796b41dc7b034f68a91a167e83106695"
format_stamp: "Formatted at 2026-06-20 14:47:20 on dist-test-slave-q9j6"
server_key: "37d8b23f5554dddde029db1709a362f4"
server_key_iv: "071234d284264c7c9483b279f8613a11"
server_key_version: "encryptionkey@0"
I20260620 14:47:20.312608 1584 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskUci3DU/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781966829091300-388-0/minicluster-data/ts-0/wal/instance:
uuid: "796b41dc7b034f68a91a167e83106695"
format_stamp: "Formatted at 2026-06-20 14:47:20 on dist-test-slave-q9j6"
server_key: "37d8b23f5554dddde029db1709a362f4"
server_key_iv: "071234d284264c7c9483b279f8613a11"
server_key_version: "encryptionkey@0"
I20260620 14:47:20.318230 1584 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260620 14:47:20.322057 1599 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 14:47:20.323555 1584 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260620 14:47:20.323788 1584 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskUci3DU/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781966829091300-388-0/minicluster-data/ts-0/data,/tmp/dist-test-taskUci3DU/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781966829091300-388-0/minicluster-data/ts-0/wal
uuid: "796b41dc7b034f68a91a167e83106695"
format_stamp: "Formatted at 2026-06-20 14:47:20 on dist-test-slave-q9j6"
server_key: "37d8b23f5554dddde029db1709a362f4"
server_key_iv: "071234d284264c7c9483b279f8613a11"
server_key_version: "encryptionkey@0"
I20260620 14:47:20.323997 1584 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskUci3DU/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781966829091300-388-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskUci3DU/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781966829091300-388-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskUci3DU/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781966829091300-388-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 14:47:20.355876 1584 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 14:47:20.360100 1584 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 14:47:20.360333 1584 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 14:47:20.363075 1584 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 14:47:20.363162 1584 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 14:47:20.363310 1584 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 14:47:20.363384 1584 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Jun 20 14:47:20 dist-test-slave-q9j6 krb5kdc[1468](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781966840, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.1@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260620 14:47:20.392442 1608 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:47:20.364072 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.1:47771 (local address 127.0.97.62:37781)
0620 14:47:20.364538 (+ 466us) server_negotiation.cc:206] Beginning negotiation
0620 14:47:20.364543 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 14:47:20.366407 (+ 1864us) server_negotiation.cc:407] Connection header received
0620 14:47:20.368824 (+ 2417us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:47:20.368830 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:47:20.368914 (+ 84us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:47:20.369041 (+ 127us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:47:20.371557 (+ 2516us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:47:20.372836 (+ 1279us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:47:20.376189 (+ 3353us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:47:20.376879 (+ 690us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:47:20.377037 (+ 158us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:47:20.383301 (+ 6264us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 14:47:20.383307 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 14:47:20.383310 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 14:47:20.383372 (+ 62us) server_negotiation.cc:922] calling sasl_server_start()
0620 14:47:20.387207 (+ 3835us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:47:20.388740 (+ 1533us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:47:20.388748 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:47:20.388752 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:47:20.388938 (+ 186us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:47:20.389625 (+ 687us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:47:20.389630 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:47:20.389635 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:47:20.390003 (+ 368us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 14:47:20.390169 (+ 166us) server_negotiation.cc:1035] Waiting for connection context
0620 14:47:20.391839 (+ 1670us) server_negotiation.cc:299] Negotiation successful
0620 14:47:20.392066 (+ 227us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":350,"thread_start_us":263,"threads_started":1}
I20260620 14:47:20.393882 1609 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:47:20.364819 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:37781 (local address 127.0.97.1:47771)
0620 14:47:20.366109 (+ 1290us) negotiation.cc:107] Waiting for socket to connect
0620 14:47:20.366265 (+ 156us) client_negotiation.cc:174] Beginning negotiation
0620 14:47:20.368399 (+ 2134us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 14:47:20.369412 (+ 1013us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 14:47:20.369430 (+ 18us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 14:47:20.370296 (+ 866us) client_negotiation.cc:189] Negotiated authn=SASL
0620 14:47:20.371311 (+ 1015us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:47:20.371343 (+ 32us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:47:20.373091 (+ 1748us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:47:20.373098 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:47:20.375973 (+ 2875us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:47:20.375991 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:47:20.377237 (+ 1246us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:47:20.377252 (+ 15us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:47:20.377695 (+ 443us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:47:20.378767 (+ 1072us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 14:47:20.378822 (+ 55us) client_negotiation.cc:656] calling sasl_client_start()
0620 14:47:20.382981 (+ 4159us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 14:47:20.387462 (+ 4481us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 14:47:20.387477 (+ 15us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 14:47:20.387511 (+ 34us) client_negotiation.cc:760] calling sasl_client_step()
0620 14:47:20.388527 (+ 1016us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 14:47:20.389095 (+ 568us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 14:47:20.389100 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 14:47:20.389104 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0620 14:47:20.389478 (+ 374us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 14:47:20.390268 (+ 790us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 14:47:20.390275 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 14:47:20.390878 (+ 603us) client_negotiation.cc:769] Sending connection context
0620 14:47:20.391285 (+ 407us) client_negotiation.cc:240] Negotiation successful
0620 14:47:20.391979 (+ 694us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1024,"thread_start_us":818,"threads_started":1}
I20260620 14:47:20.414059 1719 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.1:45961 every 8 connection(s)
I20260620 14:47:20.414089 1584 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.1:45961
I20260620 14:47:20.415933 1584 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskUci3DU/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781966829091300-388-0/minicluster-data/ts-0/data/info.pb
I20260620 14:47:20.418157 1691 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:47:20.401245 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:37781 (local address 127.0.97.1:54799)
0620 14:47:20.401932 (+ 687us) negotiation.cc:107] Waiting for socket to connect
0620 14:47:20.401948 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0620 14:47:20.402142 (+ 194us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 14:47:20.402737 (+ 595us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 14:47:20.402741 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 14:47:20.403640 (+ 899us) client_negotiation.cc:189] Negotiated authn=SASL
0620 14:47:20.404001 (+ 361us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:47:20.404011 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:47:20.405086 (+ 1075us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:47:20.405095 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:47:20.407445 (+ 2350us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:47:20.407470 (+ 25us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:47:20.408530 (+ 1060us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:47:20.408540 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:47:20.408779 (+ 239us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:47:20.409744 (+ 965us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 14:47:20.409778 (+ 34us) client_negotiation.cc:656] calling sasl_client_start()
0620 14:47:20.411270 (+ 1492us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 14:47:20.415121 (+ 3851us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 14:47:20.415125 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 14:47:20.415128 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0620 14:47:20.415562 (+ 434us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 14:47:20.416058 (+ 496us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 14:47:20.416064 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 14:47:20.416068 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0620 14:47:20.416213 (+ 145us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 14:47:20.417003 (+ 790us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 14:47:20.417008 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 14:47:20.417269 (+ 261us) client_negotiation.cc:769] Sending connection context
0620 14:47:20.417414 (+ 145us) client_negotiation.cc:240] Negotiation successful
0620 14:47:20.417795 (+ 381us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":559,"thread_start_us":410,"threads_started":1}
I20260620 14:47:20.418186 1720 heartbeater.cc:347] Connected to a master server at 127.0.97.62:37781
I20260620 14:47:20.418480 1720 heartbeater.cc:464] Registering TS with master...
I20260620 14:47:20.418972 1608 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:47:20.401135 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.1:54799 (local address 127.0.97.62:37781)
0620 14:47:20.401323 (+ 188us) server_negotiation.cc:206] Beginning negotiation
0620 14:47:20.401327 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 14:47:20.402048 (+ 721us) server_negotiation.cc:407] Connection header received
0620 14:47:20.402304 (+ 256us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:47:20.402310 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:47:20.402409 (+ 99us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:47:20.402513 (+ 104us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:47:20.404167 (+ 1654us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:47:20.404919 (+ 752us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:47:20.407688 (+ 2769us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:47:20.408285 (+ 597us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:47:20.408435 (+ 150us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:47:20.411544 (+ 3109us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 14:47:20.411549 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 14:47:20.411552 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 14:47:20.411589 (+ 37us) server_negotiation.cc:922] calling sasl_server_start()
0620 14:47:20.414955 (+ 3366us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:47:20.415731 (+ 776us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:47:20.415737 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:47:20.415741 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:47:20.415915 (+ 174us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:47:20.416349 (+ 434us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:47:20.416357 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:47:20.416361 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:47:20.416819 (+ 458us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 14:47:20.418452 (+ 1633us) server_negotiation.cc:1035] Waiting for connection context
0620 14:47:20.418573 (+ 121us) server_negotiation.cc:299] Negotiation successful
0620 14:47:20.418753 (+ 180us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":62}
I20260620 14:47:20.419262 1720 heartbeater.cc:511] Master 127.0.97.62:37781 requested a full tablet report, sending...
I20260620 14:47:20.421530 1513 ts_manager.cc:194] Registered new tserver with Master: 796b41dc7b034f68a91a167e83106695 (127.0.97.1:45961)
I20260620 14:47:20.423319 1513 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.97.1@KRBTEST.COM'} at 127.0.97.1:47771
I20260620 14:47:20.423664 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskUci3DU/build/asan/bin/kudu as pid 1584
I20260620 14:47:20.423874 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskUci3DU/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781966829091300-388-0/minicluster-data/ts-0/wal/instance
I20260620 14:47:20.424607 388 external_mini_cluster.cc:1428] Setting key 1df298157f7ef7f7ca03f13d238948de
WARNING: no policy specified for kudu/127.0.97.2@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskUci3DU/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781966829091300-388-0/krb5kdc/kudu_127.0.97.2.keytab.
Entry for principal kudu/127.0.97.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskUci3DU/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781966829091300-388-0/krb5kdc/kudu_127.0.97.2.keytab.
W20260620 14:47:20.586372 388 mini_kdc.cc:262] Time spent creating service keytab for kudu/127.0.97.2: real 0.159s user 0.000s sys 0.008s
I20260620 14:47:20.587051 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskUci3DU/build/asan/bin/kudu
/tmp/dist-test-taskUci3DU/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskUci3DU/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781966829091300-388-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskUci3DU/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781966829091300-388-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskUci3DU/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781966829091300-388-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskUci3DU/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781966829091300-388-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.2:0
--local_ip_for_outbound_sockets=127.0.97.2
--webserver_interface=127.0.97.2
--webserver_port=0
--tserver_master_addrs=127.0.97.62:37781
--keytab_file=/tmp/dist-test-taskUci3DU/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781966829091300-388-0/krb5kdc/kudu_127.0.97.2.keytab
--principal=kudu/127.0.97.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:37215
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-taskUci3DU/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781966829091300-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskUci3DU/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781966829091300-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskUci3DU/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781966829091300-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 14:47:20.925487 1724 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 14:47:20.925798 1724 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 14:47:20.925927 1724 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 14:47:20.933602 1724 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 14:47:20.933784 1724 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 14:47:20.933933 1724 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260620 14:47:20.934016 1724 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.2
I20260620 14:47:20.944371 1724 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:37215
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskUci3DU/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781966829091300-388-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskUci3DU/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781966829091300-388-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskUci3DU/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781966829091300-388-0/krb5kdc/kudu_127.0.97.2.keytab
--principal=kudu/127.0.97.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskUci3DU/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781966829091300-388-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.97.2
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.0.97.62:37781
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.2
--log_dir=/tmp/dist-test-taskUci3DU/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781966829091300-388-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 13:41:20 UTC on ccycloud.quasar-vdagoi.root.comops.site
build id 4457462
ASAN enabled
I20260620 14:47:20.945657 1724 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 14:47:20.946938 1724 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 14:47:20.960281 1732 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 14:47:20.960363 1729 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 14:47:20.960351 1730 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 14:47:20.969312 1724 server_base.cc:1034] running on GCE node
I20260620 14:47:20.970034 1724 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 14:47:20.971065 1724 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 14:47:20.972307 1724 hybrid_clock.cc:648] HybridClock initialized: now 1781966840972227 us; error 62 us; skew 500 ppm
I20260620 14:47:20.972620 1724 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 14:47:20 dist-test-slave-q9j6 krb5kdc[1468](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781966840, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 14:47:20.975880 1724 init.cc:377] Logged in from keytab as kudu/127.0.97.2@KRBTEST.COM (short username kudu)
I20260620 14:47:20.977476 1724 webserver.cc:460] Webserver started at http://127.0.97.2:36345/ using document root <none> and password file <none>
I20260620 14:47:20.978212 1724 fs_manager.cc:359] Metadata directory not provided
I20260620 14:47:20.978312 1724 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 14:47:20.978591 1724 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 14:47:20.981647 1724 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskUci3DU/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781966829091300-388-0/minicluster-data/ts-1/data/instance:
uuid: "58089d127abc4bc9b8e27348383b182d"
format_stamp: "Formatted at 2026-06-20 14:47:20 on dist-test-slave-q9j6"
server_key: "c3baad02c83c011e3bded826684de8dd"
server_key_iv: "232caef2231458da627b0e55db054447"
server_key_version: "encryptionkey@0"
I20260620 14:47:20.982462 1724 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskUci3DU/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781966829091300-388-0/minicluster-data/ts-1/wal/instance:
uuid: "58089d127abc4bc9b8e27348383b182d"
format_stamp: "Formatted at 2026-06-20 14:47:20 on dist-test-slave-q9j6"
server_key: "c3baad02c83c011e3bded826684de8dd"
server_key_iv: "232caef2231458da627b0e55db054447"
server_key_version: "encryptionkey@0"
I20260620 14:47:20.987962 1724 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.007s sys 0.000s
I20260620 14:47:20.991571 1739 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 14:47:20.993124 1724 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260620 14:47:20.993319 1724 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskUci3DU/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781966829091300-388-0/minicluster-data/ts-1/data,/tmp/dist-test-taskUci3DU/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781966829091300-388-0/minicluster-data/ts-1/wal
uuid: "58089d127abc4bc9b8e27348383b182d"
format_stamp: "Formatted at 2026-06-20 14:47:20 on dist-test-slave-q9j6"
server_key: "c3baad02c83c011e3bded826684de8dd"
server_key_iv: "232caef2231458da627b0e55db054447"
server_key_version: "encryptionkey@0"
I20260620 14:47:20.993562 1724 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskUci3DU/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781966829091300-388-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskUci3DU/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781966829091300-388-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskUci3DU/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781966829091300-388-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 14:47:21.016812 1724 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 14:47:21.021183 1724 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 14:47:21.021474 1724 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 14:47:21.024359 1724 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 14:47:21.024467 1724 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 14:47:21.024606 1724 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 14:47:21.024686 1724 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Jun 20 14:47:21 dist-test-slave-q9j6 krb5kdc[1468](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781966840, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.2@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260620 14:47:21.051019 1748 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:47:21.024897 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.2:41693 (local address 127.0.97.62:37781)
0620 14:47:21.025629 (+ 732us) server_negotiation.cc:206] Beginning negotiation
0620 14:47:21.025633 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 14:47:21.026677 (+ 1044us) server_negotiation.cc:407] Connection header received
0620 14:47:21.029086 (+ 2409us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:47:21.029092 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:47:21.029182 (+ 90us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:47:21.029307 (+ 125us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:47:21.031733 (+ 2426us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:47:21.032804 (+ 1071us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:47:21.035545 (+ 2741us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:47:21.036125 (+ 580us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:47:21.036277 (+ 152us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:47:21.042586 (+ 6309us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 14:47:21.042597 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 14:47:21.042604 (+ 7us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 14:47:21.042655 (+ 51us) server_negotiation.cc:922] calling sasl_server_start()
0620 14:47:21.045935 (+ 3280us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:47:21.047332 (+ 1397us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:47:21.047340 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:47:21.047346 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:47:21.047534 (+ 188us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:47:21.048318 (+ 784us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:47:21.048325 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:47:21.048329 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:47:21.048788 (+ 459us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 14:47:21.048958 (+ 170us) server_negotiation.cc:1035] Waiting for connection context
0620 14:47:21.050231 (+ 1273us) server_negotiation.cc:299] Negotiation successful
0620 14:47:21.050530 (+ 299us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":626,"thread_start_us":538,"threads_started":1}
I20260620 14:47:21.053365 1749 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:47:21.025669 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:37781 (local address 127.0.97.2:41693)
0620 14:47:21.026507 (+ 838us) negotiation.cc:107] Waiting for socket to connect
0620 14:47:21.026551 (+ 44us) client_negotiation.cc:174] Beginning negotiation
0620 14:47:21.028770 (+ 2219us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 14:47:21.029589 (+ 819us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 14:47:21.029607 (+ 18us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 14:47:21.030532 (+ 925us) client_negotiation.cc:189] Negotiated authn=SASL
0620 14:47:21.031440 (+ 908us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:47:21.031476 (+ 36us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:47:21.032963 (+ 1487us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:47:21.032974 (+ 11us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:47:21.035367 (+ 2393us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:47:21.035390 (+ 23us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:47:21.036500 (+ 1110us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:47:21.036506 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:47:21.036922 (+ 416us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:47:21.037926 (+ 1004us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 14:47:21.037953 (+ 27us) client_negotiation.cc:656] calling sasl_client_start()
0620 14:47:21.042266 (+ 4313us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 14:47:21.046144 (+ 3878us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 14:47:21.046159 (+ 15us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 14:47:21.046185 (+ 26us) client_negotiation.cc:760] calling sasl_client_step()
0620 14:47:21.047150 (+ 965us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 14:47:21.047745 (+ 595us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 14:47:21.047756 (+ 11us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 14:47:21.047763 (+ 7us) client_negotiation.cc:760] calling sasl_client_step()
0620 14:47:21.048160 (+ 397us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 14:47:21.048979 (+ 819us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 14:47:21.048992 (+ 13us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 14:47:21.049668 (+ 676us) client_negotiation.cc:769] Sending connection context
0620 14:47:21.050082 (+ 414us) client_negotiation.cc:240] Negotiation successful
0620 14:47:21.050910 (+ 828us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":595,"thread_start_us":370,"threads_started":1}
I20260620 14:47:21.069427 1859 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.2:45941 every 8 connection(s)
I20260620 14:47:21.069451 1724 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.2:45941
I20260620 14:47:21.070998 1724 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskUci3DU/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781966829091300-388-0/minicluster-data/ts-1/data/info.pb
I20260620 14:47:21.073697 1860 heartbeater.cc:347] Connected to a master server at 127.0.97.62:37781
I20260620 14:47:21.073936 1860 heartbeater.cc:464] Registering TS with master...
I20260620 14:47:21.074600 1860 heartbeater.cc:511] Master 127.0.97.62:37781 requested a full tablet report, sending...
I20260620 14:47:21.076017 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskUci3DU/build/asan/bin/kudu as pid 1724
I20260620 14:47:21.076182 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskUci3DU/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781966829091300-388-0/minicluster-data/ts-1/wal/instance
I20260620 14:47:21.076260 1513 ts_manager.cc:194] Registered new tserver with Master: 58089d127abc4bc9b8e27348383b182d (127.0.97.2:45941)
I20260620 14:47:21.076776 388 external_mini_cluster.cc:1428] Setting key e9908728e2162b3411f4f20c4267c2f7
I20260620 14:47:21.077371 1513 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.97.2@KRBTEST.COM'} at 127.0.97.2:41693
I20260620 14:47:21.077821 1748 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:47:21.062427 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.2:43559 (local address 127.0.97.62:37781)
0620 14:47:21.062712 (+ 285us) server_negotiation.cc:206] Beginning negotiation
0620 14:47:21.062722 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0620 14:47:21.064245 (+ 1523us) server_negotiation.cc:407] Connection header received
0620 14:47:21.064379 (+ 134us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:47:21.064387 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:47:21.064487 (+ 100us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:47:21.064618 (+ 131us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:47:21.065664 (+ 1046us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:47:21.066954 (+ 1290us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:47:21.068650 (+ 1696us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:47:21.069395 (+ 745us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:47:21.069583 (+ 188us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:47:21.071474 (+ 1891us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 14:47:21.071483 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 14:47:21.071491 (+ 8us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 14:47:21.071555 (+ 64us) server_negotiation.cc:922] calling sasl_server_start()
0620 14:47:21.074732 (+ 3177us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:47:21.075544 (+ 812us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:47:21.075550 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:47:21.075553 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:47:21.075740 (+ 187us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:47:21.076105 (+ 365us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:47:21.076112 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:47:21.076115 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:47:21.076565 (+ 450us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 14:47:21.076721 (+ 156us) server_negotiation.cc:1035] Waiting for connection context
0620 14:47:21.077183 (+ 462us) server_negotiation.cc:299] Negotiation successful
0620 14:47:21.077417 (+ 234us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":80}
I20260620 14:47:21.077868 1849 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:47:21.061780 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:37781 (local address 127.0.97.2:43559)
0620 14:47:21.064089 (+ 2309us) negotiation.cc:107] Waiting for socket to connect
0620 14:47:21.064109 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0620 14:47:21.064266 (+ 157us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 14:47:21.064670 (+ 404us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 14:47:21.064673 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 14:47:21.065120 (+ 447us) client_negotiation.cc:189] Negotiated authn=SASL
0620 14:47:21.065482 (+ 362us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:47:21.065497 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:47:21.067108 (+ 1611us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:47:21.067112 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:47:21.068472 (+ 1360us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:47:21.068485 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:47:21.069623 (+ 1138us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:47:21.069632 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:47:21.069778 (+ 146us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:47:21.070423 (+ 645us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 14:47:21.070440 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0620 14:47:21.071269 (+ 829us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 14:47:21.074871 (+ 3602us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 14:47:21.074876 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 14:47:21.074880 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0620 14:47:21.075416 (+ 536us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 14:47:21.075859 (+ 443us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 14:47:21.075864 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 14:47:21.075868 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0620 14:47:21.075996 (+ 128us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 14:47:21.076684 (+ 688us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 14:47:21.076689 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 14:47:21.076919 (+ 230us) client_negotiation.cc:769] Sending connection context
0620 14:47:21.077076 (+ 157us) client_negotiation.cc:240] Negotiation successful
0620 14:47:21.077404 (+ 328us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":2122,"thread_start_us":1974,"threads_started":1}
WARNING: no policy specified for kudu/127.0.97.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskUci3DU/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781966829091300-388-0/krb5kdc/kudu_127.0.97.3.keytab.
Entry for principal kudu/127.0.97.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskUci3DU/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781966829091300-388-0/krb5kdc/kudu_127.0.97.3.keytab.
I20260620 14:47:21.116688 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskUci3DU/build/asan/bin/kudu
/tmp/dist-test-taskUci3DU/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskUci3DU/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781966829091300-388-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskUci3DU/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781966829091300-388-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskUci3DU/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781966829091300-388-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskUci3DU/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781966829091300-388-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.3:0
--local_ip_for_outbound_sockets=127.0.97.3
--webserver_interface=127.0.97.3
--webserver_port=0
--tserver_master_addrs=127.0.97.62:37781
--keytab_file=/tmp/dist-test-taskUci3DU/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781966829091300-388-0/krb5kdc/kudu_127.0.97.3.keytab
--principal=kudu/127.0.97.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:37215
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-taskUci3DU/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781966829091300-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskUci3DU/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781966829091300-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskUci3DU/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781966829091300-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260620 14:47:21.426563 1720 heartbeater.cc:503] Master 127.0.97.62:37781 was elected leader, sending a full tablet report...
W20260620 14:47:21.453857 1864 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 14:47:21.454170 1864 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 14:47:21.454309 1864 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 14:47:21.461952 1864 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 14:47:21.462064 1864 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 14:47:21.462230 1864 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260620 14:47:21.462380 1864 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.3
I20260620 14:47:21.472565 1864 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:37215
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskUci3DU/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781966829091300-388-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskUci3DU/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781966829091300-388-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskUci3DU/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781966829091300-388-0/krb5kdc/kudu_127.0.97.3.keytab
--principal=kudu/127.0.97.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskUci3DU/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781966829091300-388-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.97.3
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.0.97.62:37781
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.3
--log_dir=/tmp/dist-test-taskUci3DU/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781966829091300-388-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 13:41:20 UTC on ccycloud.quasar-vdagoi.root.comops.site
build id 4457462
ASAN enabled
I20260620 14:47:21.473907 1864 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 14:47:21.475253 1864 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 14:47:21.488744 1869 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 14:47:21.488775 1872 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 14:47:21.488708 1870 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 14:47:21.489620 1864 server_base.cc:1034] running on GCE node
I20260620 14:47:21.490276 1864 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 14:47:21.491451 1864 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 14:47:21.492702 1864 hybrid_clock.cc:648] HybridClock initialized: now 1781966841492637 us; error 68 us; skew 500 ppm
I20260620 14:47:21.493012 1864 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 14:47:21 dist-test-slave-q9j6 krb5kdc[1468](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781966841, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 14:47:21.496347 1864 init.cc:377] Logged in from keytab as kudu/127.0.97.3@KRBTEST.COM (short username kudu)
I20260620 14:47:21.497926 1864 webserver.cc:460] Webserver started at http://127.0.97.3:43647/ using document root <none> and password file <none>
I20260620 14:47:21.498728 1864 fs_manager.cc:359] Metadata directory not provided
I20260620 14:47:21.498822 1864 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 14:47:21.499099 1864 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 14:47:21.502189 1864 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskUci3DU/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781966829091300-388-0/minicluster-data/ts-2/data/instance:
uuid: "ed946240f9ce40fbb23487b38915e4e5"
format_stamp: "Formatted at 2026-06-20 14:47:21 on dist-test-slave-q9j6"
server_key: "8cd06e075f9d7c496873b4425eff0b73"
server_key_iv: "4cdc0af30b48769b95c04b319a203d7b"
server_key_version: "encryptionkey@0"
I20260620 14:47:21.503006 1864 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskUci3DU/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781966829091300-388-0/minicluster-data/ts-2/wal/instance:
uuid: "ed946240f9ce40fbb23487b38915e4e5"
format_stamp: "Formatted at 2026-06-20 14:47:21 on dist-test-slave-q9j6"
server_key: "8cd06e075f9d7c496873b4425eff0b73"
server_key_iv: "4cdc0af30b48769b95c04b319a203d7b"
server_key_version: "encryptionkey@0"
I20260620 14:47:21.508460 1864 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.001s sys 0.006s
I20260620 14:47:21.512164 1879 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 14:47:21.513677 1864 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.002s
I20260620 14:47:21.513823 1864 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskUci3DU/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781966829091300-388-0/minicluster-data/ts-2/data,/tmp/dist-test-taskUci3DU/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781966829091300-388-0/minicluster-data/ts-2/wal
uuid: "ed946240f9ce40fbb23487b38915e4e5"
format_stamp: "Formatted at 2026-06-20 14:47:21 on dist-test-slave-q9j6"
server_key: "8cd06e075f9d7c496873b4425eff0b73"
server_key_iv: "4cdc0af30b48769b95c04b319a203d7b"
server_key_version: "encryptionkey@0"
I20260620 14:47:21.514009 1864 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskUci3DU/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781966829091300-388-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskUci3DU/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781966829091300-388-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskUci3DU/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781966829091300-388-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 14:47:21.535962 1864 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 14:47:21.540586 1864 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 14:47:21.540876 1864 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 14:47:21.543593 1864 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 14:47:21.543753 1864 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 14:47:21.543918 1864 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 14:47:21.543995 1864 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Jun 20 14:47:21 dist-test-slave-q9j6 krb5kdc[1468](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781966841, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.3@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260620 14:47:21.571920 1888 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:47:21.545118 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:37781 (local address 127.0.97.3:53035)
0620 14:47:21.546027 (+ 909us) negotiation.cc:107] Waiting for socket to connect
0620 14:47:21.546077 (+ 50us) client_negotiation.cc:174] Beginning negotiation
0620 14:47:21.548495 (+ 2418us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 14:47:21.549565 (+ 1070us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 14:47:21.549583 (+ 18us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 14:47:21.550708 (+ 1125us) client_negotiation.cc:189] Negotiated authn=SASL
0620 14:47:21.551882 (+ 1174us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:47:21.551923 (+ 41us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:47:21.553391 (+ 1468us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:47:21.553399 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:47:21.555332 (+ 1933us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:47:21.555348 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:47:21.556641 (+ 1293us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:47:21.556652 (+ 11us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:47:21.557005 (+ 353us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:47:21.557938 (+ 933us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 14:47:21.557984 (+ 46us) client_negotiation.cc:656] calling sasl_client_start()
0620 14:47:21.561868 (+ 3884us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 14:47:21.566110 (+ 4242us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 14:47:21.566121 (+ 11us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 14:47:21.566145 (+ 24us) client_negotiation.cc:760] calling sasl_client_step()
0620 14:47:21.566641 (+ 496us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 14:47:21.567143 (+ 502us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 14:47:21.567146 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 14:47:21.567149 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0620 14:47:21.567433 (+ 284us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 14:47:21.568269 (+ 836us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 14:47:21.568280 (+ 11us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 14:47:21.568838 (+ 558us) client_negotiation.cc:769] Sending connection context
0620 14:47:21.569283 (+ 445us) client_negotiation.cc:240] Negotiation successful
0620 14:47:21.569935 (+ 652us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":623,"thread_start_us":400,"threads_started":1}
I20260620 14:47:21.572535 1748 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:47:21.544535 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.3:53035 (local address 127.0.97.62:37781)
0620 14:47:21.544729 (+ 194us) server_negotiation.cc:206] Beginning negotiation
0620 14:47:21.544736 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0620 14:47:21.546226 (+ 1490us) server_negotiation.cc:407] Connection header received
0620 14:47:21.548958 (+ 2732us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:47:21.548964 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:47:21.549059 (+ 95us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:47:21.549195 (+ 136us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:47:21.552170 (+ 2975us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:47:21.553212 (+ 1042us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:47:21.555515 (+ 2303us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:47:21.556340 (+ 825us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:47:21.556537 (+ 197us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:47:21.562227 (+ 5690us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 14:47:21.562255 (+ 28us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 14:47:21.562264 (+ 9us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 14:47:21.562323 (+ 59us) server_negotiation.cc:922] calling sasl_server_start()
0620 14:47:21.565937 (+ 3614us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:47:21.566800 (+ 863us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:47:21.566809 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:47:21.566816 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:47:21.566986 (+ 170us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:47:21.567633 (+ 647us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:47:21.567641 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:47:21.567648 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:47:21.568106 (+ 458us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 14:47:21.568284 (+ 178us) server_negotiation.cc:1035] Waiting for connection context
0620 14:47:21.571890 (+ 3606us) server_negotiation.cc:299] Negotiation successful
0620 14:47:21.572194 (+ 304us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":66}
I20260620 14:47:21.585439 1864 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.3:45431
I20260620 14:47:21.585458 1998 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.3:45431 every 8 connection(s)
I20260620 14:47:21.586982 1864 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskUci3DU/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781966829091300-388-0/minicluster-data/ts-2/data/info.pb
I20260620 14:47:21.589498 1999 heartbeater.cc:347] Connected to a master server at 127.0.97.62:37781
I20260620 14:47:21.589757 1999 heartbeater.cc:464] Registering TS with master...
I20260620 14:47:21.590636 1999 heartbeater.cc:511] Master 127.0.97.62:37781 requested a full tablet report, sending...
I20260620 14:47:21.592363 1512 ts_manager.cc:194] Registered new tserver with Master: ed946240f9ce40fbb23487b38915e4e5 (127.0.97.3:45431)
I20260620 14:47:21.593266 1512 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.97.3@KRBTEST.COM'} at 127.0.97.3:53035
I20260620 14:47:21.594168 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskUci3DU/build/asan/bin/kudu as pid 1864
I20260620 14:47:21.594337 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskUci3DU/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781966829091300-388-0/minicluster-data/ts-2/wal/instance
I20260620 14:47:21.594887 388 external_mini_cluster.cc:1428] Setting key a6fa442d75b7566342599e6874d52159
I20260620 14:47:21.595322 1748 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:47:21.579328 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.3:34925 (local address 127.0.97.62:37781)
0620 14:47:21.579521 (+ 193us) server_negotiation.cc:206] Beginning negotiation
0620 14:47:21.579530 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0620 14:47:21.580667 (+ 1137us) server_negotiation.cc:407] Connection header received
0620 14:47:21.580902 (+ 235us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:47:21.580908 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:47:21.580979 (+ 71us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:47:21.581079 (+ 100us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:47:21.582712 (+ 1633us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:47:21.583729 (+ 1017us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:47:21.585969 (+ 2240us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:47:21.586397 (+ 428us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:47:21.586501 (+ 104us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:47:21.589069 (+ 2568us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 14:47:21.589077 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 14:47:21.589083 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 14:47:21.589124 (+ 41us) server_negotiation.cc:922] calling sasl_server_start()
0620 14:47:21.592442 (+ 3318us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:47:21.593138 (+ 696us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:47:21.593144 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:47:21.593148 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:47:21.593296 (+ 148us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:47:21.593730 (+ 434us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:47:21.593735 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:47:21.593737 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:47:21.594080 (+ 343us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 14:47:21.594257 (+ 177us) server_negotiation.cc:1035] Waiting for connection context
0620 14:47:21.594687 (+ 430us) server_negotiation.cc:299] Negotiation successful
0620 14:47:21.594908 (+ 221us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":56}
I20260620 14:47:21.595448 1985 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:47:21.579625 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:37781 (local address 127.0.97.3:34925)
0620 14:47:21.580556 (+ 931us) negotiation.cc:107] Waiting for socket to connect
0620 14:47:21.580577 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0620 14:47:21.580756 (+ 179us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 14:47:21.581111 (+ 355us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 14:47:21.581115 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 14:47:21.581664 (+ 549us) client_negotiation.cc:189] Negotiated authn=SASL
0620 14:47:21.582509 (+ 845us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:47:21.582526 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:47:21.583887 (+ 1361us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:47:21.583896 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:47:21.585803 (+ 1907us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:47:21.585819 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:47:21.586602 (+ 783us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:47:21.586608 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:47:21.586784 (+ 176us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:47:21.587574 (+ 790us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 14:47:21.587631 (+ 57us) client_negotiation.cc:656] calling sasl_client_start()
0620 14:47:21.588899 (+ 1268us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 14:47:21.592641 (+ 3742us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 14:47:21.592645 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 14:47:21.592648 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0620 14:47:21.593008 (+ 360us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 14:47:21.593413 (+ 405us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 14:47:21.593418 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 14:47:21.593422 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0620 14:47:21.593560 (+ 138us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 14:47:21.594223 (+ 663us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 14:47:21.594228 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 14:47:21.594420 (+ 192us) client_negotiation.cc:769] Sending connection context
0620 14:47:21.594587 (+ 167us) client_negotiation.cc:240] Negotiation successful
0620 14:47:21.594935 (+ 348us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":744,"thread_start_us":551,"threads_started":1}
I20260620 14:47:21.598322 388 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
Jun 20 14:47:21 dist-test-slave-q9j6 krb5kdc[1468](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781966841, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-user@KRBTEST.COM:
Jun 20 14:47:21 dist-test-slave-q9j6 krb5kdc[1468](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781966841, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260620 14:47:21.643679 1748 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:47:21.629991 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56570 (local address 127.0.97.62:37781)
0620 14:47:21.630163 (+ 172us) server_negotiation.cc:206] Beginning negotiation
0620 14:47:21.630168 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 14:47:21.630331 (+ 163us) server_negotiation.cc:407] Connection header received
0620 14:47:21.630484 (+ 153us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:47:21.630489 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:47:21.630549 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:47:21.630641 (+ 92us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:47:21.632074 (+ 1433us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:47:21.632839 (+ 765us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:47:21.634121 (+ 1282us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:47:21.634522 (+ 401us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:47:21.634613 (+ 91us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:47:21.638501 (+ 3888us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 14:47:21.638506 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 14:47:21.638510 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 14:47:21.638545 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0620 14:47:21.641267 (+ 2722us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:47:21.641999 (+ 732us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:47:21.642002 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:47:21.642005 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:47:21.642130 (+ 125us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:47:21.642465 (+ 335us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:47:21.642473 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:47:21.642479 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:47:21.642762 (+ 283us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 14:47:21.642955 (+ 193us) server_negotiation.cc:1035] Waiting for connection context
0620 14:47:21.643202 (+ 247us) server_negotiation.cc:299] Negotiation successful
0620 14:47:21.643374 (+ 172us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":57}
I20260620 14:47:21.647748 1513 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:56570:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260620 14:47:21.654445 1513 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260620 14:47:21.694895 2016 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:47:21.666614 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.2:45941 (local address 127.0.0.1:47260)
0620 14:47:21.684357 (+ 17743us) negotiation.cc:107] Waiting for socket to connect
0620 14:47:21.684406 (+ 49us) client_negotiation.cc:174] Beginning negotiation
0620 14:47:21.684612 (+ 206us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 14:47:21.686943 (+ 2331us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 14:47:21.686952 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 14:47:21.687014 (+ 62us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 14:47:21.687806 (+ 792us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:47:21.687821 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:47:21.689284 (+ 1463us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:47:21.689292 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:47:21.691851 (+ 2559us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:47:21.691866 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:47:21.693426 (+ 1560us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:47:21.693432 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:47:21.693636 (+ 204us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:47:21.693691 (+ 55us) client_negotiation.cc:769] Sending connection context
0620 14:47:21.694179 (+ 488us) client_negotiation.cc:240] Negotiation successful
0620 14:47:21.694395 (+ 216us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":17491,"mutex_wait_us":304,"spinlock_wait_cycles":19328,"thread_start_us":19888,"threads_started":1}
I20260620 14:47:21.695533 2015 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:47:21.667290 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41162 (local address 127.0.97.3:45431)
0620 14:47:21.683693 (+ 16403us) server_negotiation.cc:206] Beginning negotiation
0620 14:47:21.683707 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0620 14:47:21.685844 (+ 2137us) server_negotiation.cc:407] Connection header received
0620 14:47:21.686029 (+ 185us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:47:21.686041 (+ 12us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:47:21.686365 (+ 324us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:47:21.686571 (+ 206us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 14:47:21.687883 (+ 1312us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:47:21.689532 (+ 1649us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:47:21.692615 (+ 3083us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:47:21.694081 (+ 1466us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:47:21.694208 (+ 127us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:47:21.694444 (+ 236us) server_negotiation.cc:1035] Waiting for connection context
0620 14:47:21.695006 (+ 562us) server_negotiation.cc:299] Negotiation successful
0620 14:47:21.695187 (+ 181us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":16154,"thread_start_us":15785,"threads_started":1}
I20260620 14:47:21.695529 2013 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:47:21.666758 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47260 (local address 127.0.97.2:45941)
0620 14:47:21.686328 (+ 19570us) server_negotiation.cc:206] Beginning negotiation
0620 14:47:21.686337 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0620 14:47:21.686365 (+ 28us) server_negotiation.cc:407] Connection header received
0620 14:47:21.686456 (+ 91us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:47:21.686466 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:47:21.686723 (+ 257us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:47:21.687105 (+ 382us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 14:47:21.688014 (+ 909us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:47:21.689129 (+ 1115us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:47:21.692008 (+ 2879us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:47:21.693207 (+ 1199us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:47:21.694766 (+ 1559us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:47:21.694976 (+ 210us) server_negotiation.cc:1035] Waiting for connection context
0620 14:47:21.695072 (+ 96us) server_negotiation.cc:299] Negotiation successful
0620 14:47:21.695233 (+ 161us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":19424,"thread_start_us":19313,"threads_started":1}
I20260620 14:47:21.695639 2014 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:47:21.666717 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.3:45431 (local address 127.0.0.1:41162)
0620 14:47:21.685700 (+ 18983us) negotiation.cc:107] Waiting for socket to connect
0620 14:47:21.685716 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0620 14:47:21.685878 (+ 162us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 14:47:21.686943 (+ 1065us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 14:47:21.686952 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 14:47:21.687014 (+ 62us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 14:47:21.687580 (+ 566us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:47:21.687633 (+ 53us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:47:21.689678 (+ 2045us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:47:21.689684 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:47:21.692430 (+ 2746us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:47:21.692450 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:47:21.694300 (+ 1850us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:47:21.694307 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:47:21.694457 (+ 150us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:47:21.694486 (+ 29us) client_negotiation.cc:769] Sending connection context
0620 14:47:21.694573 (+ 87us) client_negotiation.cc:240] Negotiation successful
0620 14:47:21.695236 (+ 663us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":18841,"mutex_wait_us":57,"spinlock_wait_cycles":18688,"thread_start_us":18684,"threads_started":1}
I20260620 14:47:21.696486 2012 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:47:21.668555 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:45961 (local address 127.0.0.1:56206)
0620 14:47:21.686765 (+ 18210us) negotiation.cc:107] Waiting for socket to connect
0620 14:47:21.686782 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0620 14:47:21.686999 (+ 217us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 14:47:21.687841 (+ 842us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 14:47:21.687845 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 14:47:21.687873 (+ 28us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 14:47:21.688285 (+ 412us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:47:21.688296 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:47:21.690020 (+ 1724us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:47:21.690028 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:47:21.693106 (+ 3078us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:47:21.693117 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:47:21.694689 (+ 1572us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:47:21.694695 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:47:21.694886 (+ 191us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:47:21.695802 (+ 916us) client_negotiation.cc:769] Sending connection context
0620 14:47:21.695956 (+ 154us) client_negotiation.cc:240] Negotiation successful
0620 14:47:21.696093 (+ 137us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":18088,"mutex_wait_us":890,"spinlock_wait_cycles":3328,"thread_start_us":15414,"threads_started":1}
I20260620 14:47:21.696580 2017 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:47:21.669100 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56206 (local address 127.0.97.1:45961)
0620 14:47:21.685204 (+ 16104us) server_negotiation.cc:206] Beginning negotiation
0620 14:47:21.685215 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0620 14:47:21.686887 (+ 1672us) server_negotiation.cc:407] Connection header received
0620 14:47:21.687196 (+ 309us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:47:21.687204 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:47:21.687451 (+ 247us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:47:21.687640 (+ 189us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 14:47:21.688469 (+ 829us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:47:21.689880 (+ 1411us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:47:21.693264 (+ 3384us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:47:21.694209 (+ 945us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:47:21.694342 (+ 133us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:47:21.694548 (+ 206us) server_negotiation.cc:1035] Waiting for connection context
0620 14:47:21.696017 (+ 1469us) server_negotiation.cc:299] Negotiation successful
0620 14:47:21.696216 (+ 199us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":15952,"thread_start_us":15830,"threads_started":1}
I20260620 14:47:21.699554 1790 tablet_service.cc:1467] Processing CreateTablet for tablet db47670da1ec4462873caecda63e31b9 (DEFAULT_TABLE table=test-table [id=fad12891c23e4d8fbe518ad05366785c]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 14:47:21.699759 1929 tablet_service.cc:1467] Processing CreateTablet for tablet db47670da1ec4462873caecda63e31b9 (DEFAULT_TABLE table=test-table [id=fad12891c23e4d8fbe518ad05366785c]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 14:47:21.699895 1650 tablet_service.cc:1467] Processing CreateTablet for tablet db47670da1ec4462873caecda63e31b9 (DEFAULT_TABLE table=test-table [id=fad12891c23e4d8fbe518ad05366785c]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 14:47:21.701097 1650 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet db47670da1ec4462873caecda63e31b9. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 14:47:21.701103 1929 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet db47670da1ec4462873caecda63e31b9. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 14:47:21.701107 1790 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet db47670da1ec4462873caecda63e31b9. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 14:47:21.713323 2018 tablet_bootstrap.cc:492] T db47670da1ec4462873caecda63e31b9 P 58089d127abc4bc9b8e27348383b182d: Bootstrap starting.
I20260620 14:47:21.716404 2020 tablet_bootstrap.cc:492] T db47670da1ec4462873caecda63e31b9 P ed946240f9ce40fbb23487b38915e4e5: Bootstrap starting.
I20260620 14:47:21.716950 2018 tablet_bootstrap.cc:654] T db47670da1ec4462873caecda63e31b9 P 58089d127abc4bc9b8e27348383b182d: Neither blocks nor log segments found. Creating new log.
I20260620 14:47:21.717212 2019 tablet_bootstrap.cc:492] T db47670da1ec4462873caecda63e31b9 P 796b41dc7b034f68a91a167e83106695: Bootstrap starting.
I20260620 14:47:21.719852 2018 log.cc:824] T db47670da1ec4462873caecda63e31b9 P 58089d127abc4bc9b8e27348383b182d: Log is configured to *not* fsync() on all Append() calls
I20260620 14:47:21.720984 2020 tablet_bootstrap.cc:654] T db47670da1ec4462873caecda63e31b9 P ed946240f9ce40fbb23487b38915e4e5: Neither blocks nor log segments found. Creating new log.
I20260620 14:47:21.721122 2019 tablet_bootstrap.cc:654] T db47670da1ec4462873caecda63e31b9 P 796b41dc7b034f68a91a167e83106695: Neither blocks nor log segments found. Creating new log.
I20260620 14:47:21.722116 2020 log.cc:824] T db47670da1ec4462873caecda63e31b9 P ed946240f9ce40fbb23487b38915e4e5: Log is configured to *not* fsync() on all Append() calls
I20260620 14:47:21.722422 2019 log.cc:824] T db47670da1ec4462873caecda63e31b9 P 796b41dc7b034f68a91a167e83106695: Log is configured to *not* fsync() on all Append() calls
I20260620 14:47:21.723425 2018 tablet_bootstrap.cc:492] T db47670da1ec4462873caecda63e31b9 P 58089d127abc4bc9b8e27348383b182d: No bootstrap required, opened a new log
I20260620 14:47:21.723771 2018 ts_tablet_manager.cc:1397] T db47670da1ec4462873caecda63e31b9 P 58089d127abc4bc9b8e27348383b182d: Time spent bootstrapping tablet: real 0.011s user 0.005s sys 0.003s
I20260620 14:47:21.724972 2020 tablet_bootstrap.cc:492] T db47670da1ec4462873caecda63e31b9 P ed946240f9ce40fbb23487b38915e4e5: No bootstrap required, opened a new log
I20260620 14:47:21.725245 2020 ts_tablet_manager.cc:1397] T db47670da1ec4462873caecda63e31b9 P ed946240f9ce40fbb23487b38915e4e5: Time spent bootstrapping tablet: real 0.009s user 0.006s sys 0.000s
I20260620 14:47:21.730235 2019 tablet_bootstrap.cc:492] T db47670da1ec4462873caecda63e31b9 P 796b41dc7b034f68a91a167e83106695: No bootstrap required, opened a new log
I20260620 14:47:21.730535 2019 ts_tablet_manager.cc:1397] T db47670da1ec4462873caecda63e31b9 P 796b41dc7b034f68a91a167e83106695: Time spent bootstrapping tablet: real 0.014s user 0.007s sys 0.000s
I20260620 14:47:21.733085 2020 raft_consensus.cc:348] T db47670da1ec4462873caecda63e31b9 P ed946240f9ce40fbb23487b38915e4e5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "58089d127abc4bc9b8e27348383b182d" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 45941 } } peers { permanent_uuid: "ed946240f9ce40fbb23487b38915e4e5" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 45431 } } peers { permanent_uuid: "796b41dc7b034f68a91a167e83106695" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 45961 } }
I20260620 14:47:21.733477 2020 raft_consensus.cc:374] T db47670da1ec4462873caecda63e31b9 P ed946240f9ce40fbb23487b38915e4e5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 14:47:21.733594 2020 raft_consensus.cc:729] T db47670da1ec4462873caecda63e31b9 P ed946240f9ce40fbb23487b38915e4e5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ed946240f9ce40fbb23487b38915e4e5, State: Initialized, Role: FOLLOWER
I20260620 14:47:21.734125 2018 raft_consensus.cc:348] T db47670da1ec4462873caecda63e31b9 P 58089d127abc4bc9b8e27348383b182d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "58089d127abc4bc9b8e27348383b182d" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 45941 } } peers { permanent_uuid: "ed946240f9ce40fbb23487b38915e4e5" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 45431 } } peers { permanent_uuid: "796b41dc7b034f68a91a167e83106695" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 45961 } }
I20260620 14:47:21.734335 2020 consensus_queue.cc:260] T db47670da1ec4462873caecda63e31b9 P ed946240f9ce40fbb23487b38915e4e5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "58089d127abc4bc9b8e27348383b182d" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 45941 } } peers { permanent_uuid: "ed946240f9ce40fbb23487b38915e4e5" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 45431 } } peers { permanent_uuid: "796b41dc7b034f68a91a167e83106695" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 45961 } }
I20260620 14:47:21.734546 2018 raft_consensus.cc:374] T db47670da1ec4462873caecda63e31b9 P 58089d127abc4bc9b8e27348383b182d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 14:47:21.734673 2018 raft_consensus.cc:729] T db47670da1ec4462873caecda63e31b9 P 58089d127abc4bc9b8e27348383b182d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 58089d127abc4bc9b8e27348383b182d, State: Initialized, Role: FOLLOWER
I20260620 14:47:21.735288 2018 consensus_queue.cc:260] T db47670da1ec4462873caecda63e31b9 P 58089d127abc4bc9b8e27348383b182d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "58089d127abc4bc9b8e27348383b182d" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 45941 } } peers { permanent_uuid: "ed946240f9ce40fbb23487b38915e4e5" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 45431 } } peers { permanent_uuid: "796b41dc7b034f68a91a167e83106695" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 45961 } }
I20260620 14:47:21.736557 1860 heartbeater.cc:503] Master 127.0.97.62:37781 was elected leader, sending a full tablet report...
I20260620 14:47:21.737178 2018 ts_tablet_manager.cc:1428] T db47670da1ec4462873caecda63e31b9 P 58089d127abc4bc9b8e27348383b182d: Time spent starting tablet: real 0.013s user 0.012s sys 0.000s
I20260620 14:47:21.738535 1999 heartbeater.cc:503] Master 127.0.97.62:37781 was elected leader, sending a full tablet report...
I20260620 14:47:21.739188 2020 ts_tablet_manager.cc:1428] T db47670da1ec4462873caecda63e31b9 P ed946240f9ce40fbb23487b38915e4e5: Time spent starting tablet: real 0.014s user 0.014s sys 0.000s
I20260620 14:47:21.741735 2019 raft_consensus.cc:348] T db47670da1ec4462873caecda63e31b9 P 796b41dc7b034f68a91a167e83106695 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "58089d127abc4bc9b8e27348383b182d" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 45941 } } peers { permanent_uuid: "ed946240f9ce40fbb23487b38915e4e5" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 45431 } } peers { permanent_uuid: "796b41dc7b034f68a91a167e83106695" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 45961 } }
I20260620 14:47:21.742268 2019 raft_consensus.cc:374] T db47670da1ec4462873caecda63e31b9 P 796b41dc7b034f68a91a167e83106695 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 14:47:21.742384 2019 raft_consensus.cc:729] T db47670da1ec4462873caecda63e31b9 P 796b41dc7b034f68a91a167e83106695 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 796b41dc7b034f68a91a167e83106695, State: Initialized, Role: FOLLOWER
I20260620 14:47:21.743125 2019 consensus_queue.cc:260] T db47670da1ec4462873caecda63e31b9 P 796b41dc7b034f68a91a167e83106695 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "58089d127abc4bc9b8e27348383b182d" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 45941 } } peers { permanent_uuid: "ed946240f9ce40fbb23487b38915e4e5" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 45431 } } peers { permanent_uuid: "796b41dc7b034f68a91a167e83106695" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 45961 } }
I20260620 14:47:21.744642 2019 ts_tablet_manager.cc:1428] T db47670da1ec4462873caecda63e31b9 P 796b41dc7b034f68a91a167e83106695: Time spent starting tablet: real 0.014s user 0.014s sys 0.001s
W20260620 14:47:21.823348 1861 tablet.cc:2369] T db47670da1ec4462873caecda63e31b9 P 58089d127abc4bc9b8e27348383b182d: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260620 14:47:21.839041 2000 tablet.cc:2369] T db47670da1ec4462873caecda63e31b9 P ed946240f9ce40fbb23487b38915e4e5: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260620 14:47:21.879400 2025 raft_consensus.cc:482] T db47670da1ec4462873caecda63e31b9 P 58089d127abc4bc9b8e27348383b182d [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260620 14:47:21.879741 2025 raft_consensus.cc:504] T db47670da1ec4462873caecda63e31b9 P 58089d127abc4bc9b8e27348383b182d [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "58089d127abc4bc9b8e27348383b182d" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 45941 } } peers { permanent_uuid: "ed946240f9ce40fbb23487b38915e4e5" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 45431 } } peers { permanent_uuid: "796b41dc7b034f68a91a167e83106695" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 45961 } }
I20260620 14:47:21.881654 2025 leader_election.cc:288] T db47670da1ec4462873caecda63e31b9 P 58089d127abc4bc9b8e27348383b182d [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers ed946240f9ce40fbb23487b38915e4e5 (127.0.97.3:45431), 796b41dc7b034f68a91a167e83106695 (127.0.97.1:45961)
I20260620 14:47:21.891286 2015 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:47:21.882036 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.2:48775 (local address 127.0.97.3:45431)
0620 14:47:21.882246 (+ 210us) server_negotiation.cc:206] Beginning negotiation
0620 14:47:21.882256 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0620 14:47:21.883014 (+ 758us) server_negotiation.cc:407] Connection header received
0620 14:47:21.883186 (+ 172us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:47:21.883191 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:47:21.883280 (+ 89us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:47:21.883407 (+ 127us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 14:47:21.884106 (+ 699us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:47:21.885220 (+ 1114us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:47:21.888826 (+ 3606us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:47:21.890071 (+ 1245us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:47:21.890221 (+ 150us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:47:21.890331 (+ 110us) server_negotiation.cc:1035] Waiting for connection context
0620 14:47:21.890879 (+ 548us) server_negotiation.cc:299] Negotiation successful
0620 14:47:21.891016 (+ 137us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":75}
I20260620 14:47:21.891283 2017 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:47:21.882141 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.2:57791 (local address 127.0.97.1:45961)
0620 14:47:21.882382 (+ 241us) server_negotiation.cc:206] Beginning negotiation
0620 14:47:21.882390 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0620 14:47:21.883161 (+ 771us) server_negotiation.cc:407] Connection header received
0620 14:47:21.883317 (+ 156us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:47:21.883322 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:47:21.883406 (+ 84us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:47:21.883493 (+ 87us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 14:47:21.884246 (+ 753us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:47:21.885263 (+ 1017us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:47:21.888825 (+ 3562us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:47:21.890071 (+ 1246us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:47:21.890221 (+ 150us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:47:21.890331 (+ 110us) server_negotiation.cc:1035] Waiting for connection context
0620 14:47:21.890879 (+ 548us) server_negotiation.cc:299] Negotiation successful
0620 14:47:21.891016 (+ 137us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":89}
I20260620 14:47:21.891506 2028 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:47:21.882105 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:45961 (local address 127.0.97.2:57791)
0620 14:47:21.883019 (+ 914us) negotiation.cc:107] Waiting for socket to connect
0620 14:47:21.883037 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0620 14:47:21.883173 (+ 136us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 14:47:21.883630 (+ 457us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 14:47:21.883634 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 14:47:21.883660 (+ 26us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 14:47:21.884093 (+ 433us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:47:21.884104 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:47:21.885385 (+ 1281us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:47:21.885389 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:47:21.888639 (+ 3250us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:47:21.888657 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:47:21.890373 (+ 1716us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:47:21.890379 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:47:21.890640 (+ 261us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:47:21.890735 (+ 95us) client_negotiation.cc:769] Sending connection context
0620 14:47:21.890836 (+ 101us) client_negotiation.cc:240] Negotiation successful
0620 14:47:21.891023 (+ 187us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":761,"mutex_wait_us":235,"thread_start_us":533,"threads_started":1}
I20260620 14:47:21.891533 2027 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:47:21.882038 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.3:45431 (local address 127.0.97.2:48775)
0620 14:47:21.882894 (+ 856us) negotiation.cc:107] Waiting for socket to connect
0620 14:47:21.882910 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0620 14:47:21.883037 (+ 127us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 14:47:21.883422 (+ 385us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 14:47:21.883427 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 14:47:21.883458 (+ 31us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 14:47:21.883956 (+ 498us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:47:21.883970 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:47:21.885384 (+ 1414us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:47:21.885389 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:47:21.888639 (+ 3250us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:47:21.888657 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:47:21.890373 (+ 1716us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:47:21.890379 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:47:21.890638 (+ 259us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:47:21.890735 (+ 97us) client_negotiation.cc:769] Sending connection context
0620 14:47:21.890836 (+ 101us) client_negotiation.cc:240] Negotiation successful
0620 14:47:21.891023 (+ 187us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":714,"mutex_wait_us":342,"thread_start_us":469,"threads_started":1}
I20260620 14:47:21.892581 1670 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "db47670da1ec4462873caecda63e31b9" candidate_uuid: "58089d127abc4bc9b8e27348383b182d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "796b41dc7b034f68a91a167e83106695" is_pre_election: true
I20260620 14:47:21.892786 1949 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "db47670da1ec4462873caecda63e31b9" candidate_uuid: "58089d127abc4bc9b8e27348383b182d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ed946240f9ce40fbb23487b38915e4e5" is_pre_election: true
I20260620 14:47:21.893186 1670 raft_consensus.cc:2454] T db47670da1ec4462873caecda63e31b9 P 796b41dc7b034f68a91a167e83106695 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 58089d127abc4bc9b8e27348383b182d in term 0.
I20260620 14:47:21.893321 1949 raft_consensus.cc:2454] T db47670da1ec4462873caecda63e31b9 P ed946240f9ce40fbb23487b38915e4e5 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 58089d127abc4bc9b8e27348383b182d in term 0.
I20260620 14:47:21.893986 1742 leader_election.cc:302] T db47670da1ec4462873caecda63e31b9 P 58089d127abc4bc9b8e27348383b182d [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 3 yes votes; 0 no votes. yes voters: 58089d127abc4bc9b8e27348383b182d, 796b41dc7b034f68a91a167e83106695, ed946240f9ce40fbb23487b38915e4e5; no voters:
I20260620 14:47:21.894455 2025 raft_consensus.cc:2781] T db47670da1ec4462873caecda63e31b9 P 58089d127abc4bc9b8e27348383b182d [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260620 14:47:21.894610 2025 raft_consensus.cc:482] T db47670da1ec4462873caecda63e31b9 P 58089d127abc4bc9b8e27348383b182d [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260620 14:47:21.894704 2025 raft_consensus.cc:3037] T db47670da1ec4462873caecda63e31b9 P 58089d127abc4bc9b8e27348383b182d [term 0 FOLLOWER]: Advancing to term 1
I20260620 14:47:21.897011 2025 raft_consensus.cc:504] T db47670da1ec4462873caecda63e31b9 P 58089d127abc4bc9b8e27348383b182d [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "58089d127abc4bc9b8e27348383b182d" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 45941 } } peers { permanent_uuid: "ed946240f9ce40fbb23487b38915e4e5" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 45431 } } peers { permanent_uuid: "796b41dc7b034f68a91a167e83106695" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 45961 } }
I20260620 14:47:21.897684 2025 leader_election.cc:288] T db47670da1ec4462873caecda63e31b9 P 58089d127abc4bc9b8e27348383b182d [CANDIDATE]: Term 1 election: Requested vote from peers ed946240f9ce40fbb23487b38915e4e5 (127.0.97.3:45431), 796b41dc7b034f68a91a167e83106695 (127.0.97.1:45961)
I20260620 14:47:21.898125 1949 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "db47670da1ec4462873caecda63e31b9" candidate_uuid: "58089d127abc4bc9b8e27348383b182d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ed946240f9ce40fbb23487b38915e4e5"
I20260620 14:47:21.898200 1670 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "db47670da1ec4462873caecda63e31b9" candidate_uuid: "58089d127abc4bc9b8e27348383b182d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "796b41dc7b034f68a91a167e83106695"
I20260620 14:47:21.898330 1949 raft_consensus.cc:3037] T db47670da1ec4462873caecda63e31b9 P ed946240f9ce40fbb23487b38915e4e5 [term 0 FOLLOWER]: Advancing to term 1
I20260620 14:47:21.898461 1670 raft_consensus.cc:3037] T db47670da1ec4462873caecda63e31b9 P 796b41dc7b034f68a91a167e83106695 [term 0 FOLLOWER]: Advancing to term 1
I20260620 14:47:21.900718 1670 raft_consensus.cc:2454] T db47670da1ec4462873caecda63e31b9 P 796b41dc7b034f68a91a167e83106695 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 58089d127abc4bc9b8e27348383b182d in term 1.
I20260620 14:47:21.901037 1949 raft_consensus.cc:2454] T db47670da1ec4462873caecda63e31b9 P ed946240f9ce40fbb23487b38915e4e5 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 58089d127abc4bc9b8e27348383b182d in term 1.
I20260620 14:47:21.901131 1742 leader_election.cc:302] T db47670da1ec4462873caecda63e31b9 P 58089d127abc4bc9b8e27348383b182d [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 58089d127abc4bc9b8e27348383b182d, 796b41dc7b034f68a91a167e83106695; no voters:
I20260620 14:47:21.901443 2025 raft_consensus.cc:2781] T db47670da1ec4462873caecda63e31b9 P 58089d127abc4bc9b8e27348383b182d [term 1 FOLLOWER]: Leader election won for term 1
I20260620 14:47:21.901877 2025 raft_consensus.cc:686] T db47670da1ec4462873caecda63e31b9 P 58089d127abc4bc9b8e27348383b182d [term 1 LEADER]: Becoming Leader. State: Replica: 58089d127abc4bc9b8e27348383b182d, State: Running, Role: LEADER
I20260620 14:47:21.902364 2025 consensus_queue.cc:237] T db47670da1ec4462873caecda63e31b9 P 58089d127abc4bc9b8e27348383b182d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "58089d127abc4bc9b8e27348383b182d" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 45941 } } peers { permanent_uuid: "ed946240f9ce40fbb23487b38915e4e5" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 45431 } } peers { permanent_uuid: "796b41dc7b034f68a91a167e83106695" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 45961 } }
I20260620 14:47:21.907486 1513 catalog_manager.cc:5495] T db47670da1ec4462873caecda63e31b9 P 58089d127abc4bc9b8e27348383b182d reported cstate change: term changed from 0 to 1, leader changed from <none> to 58089d127abc4bc9b8e27348383b182d (127.0.97.2). New cstate: current_term: 1 leader_uuid: "58089d127abc4bc9b8e27348383b182d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "58089d127abc4bc9b8e27348383b182d" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 45941 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "ed946240f9ce40fbb23487b38915e4e5" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 45431 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "796b41dc7b034f68a91a167e83106695" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 45961 } health_report { overall_health: UNKNOWN } } }
W20260620 14:47:21.918764 1721 tablet.cc:2369] T db47670da1ec4462873caecda63e31b9 P 796b41dc7b034f68a91a167e83106695: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260620 14:47:21.945218 2013 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:47:21.940125 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47266 (local address 127.0.97.2:45941)
0620 14:47:21.940304 (+ 179us) server_negotiation.cc:206] Beginning negotiation
0620 14:47:21.940308 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 14:47:21.940327 (+ 19us) server_negotiation.cc:407] Connection header received
0620 14:47:21.940607 (+ 280us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:47:21.940611 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:47:21.940688 (+ 77us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:47:21.940819 (+ 131us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 14:47:21.941139 (+ 320us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:47:21.941888 (+ 749us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:47:21.943213 (+ 1325us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:47:21.943658 (+ 445us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:47:21.943762 (+ 104us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:47:21.944176 (+ 414us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 14:47:21.944731 (+ 555us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0620 14:47:21.944845 (+ 114us) server_negotiation.cc:1035] Waiting for connection context
0620 14:47:21.944950 (+ 105us) server_negotiation.cc:299] Negotiation successful
0620 14:47:21.945016 (+ 66us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":58}
I20260620 14:47:21.953619 1670 raft_consensus.cc:1261] T db47670da1ec4462873caecda63e31b9 P 796b41dc7b034f68a91a167e83106695 [term 1 FOLLOWER]: Refusing update from remote peer 58089d127abc4bc9b8e27348383b182d: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 14:47:21.953619 1949 raft_consensus.cc:1261] T db47670da1ec4462873caecda63e31b9 P ed946240f9ce40fbb23487b38915e4e5 [term 1 FOLLOWER]: Refusing update from remote peer 58089d127abc4bc9b8e27348383b182d: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 14:47:21.954572 2029 consensus_queue.cc:1035] T db47670da1ec4462873caecda63e31b9 P 58089d127abc4bc9b8e27348383b182d [LEADER]: Connected to new peer: Peer: permanent_uuid: "ed946240f9ce40fbb23487b38915e4e5" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 45431 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 14:47:21.954991 2025 consensus_queue.cc:1035] T db47670da1ec4462873caecda63e31b9 P 58089d127abc4bc9b8e27348383b182d [LEADER]: Connected to new peer: Peer: permanent_uuid: "796b41dc7b034f68a91a167e83106695" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 45961 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 14:47:21.966818 2034 mvcc.cc:205] Tried to move back new op lower bound from 7298936184632283136 to 7298936184435560448. Current Snapshot: MvccSnapshot[applied={T|T < 7298936184632283136}]
I20260620 14:47:21.985419 1513 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:56570:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 14:47:21.985723 1513 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:56570:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 14:47:21.989331 1513 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 8f6b9a65e0374b0b9f1f530836785c11: Sending DeleteTablet for 3 replicas of tablet db47670da1ec4462873caecda63e31b9
I20260620 14:47:21.990535 1929 tablet_service.cc:1514] Processing DeleteTablet for tablet db47670da1ec4462873caecda63e31b9 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 14:47:21 UTC) from {username='kudu', principal='kudu/127.0.97.62@KRBTEST.COM'} at 127.0.0.1:41162
I20260620 14:47:21.990543 1790 tablet_service.cc:1514] Processing DeleteTablet for tablet db47670da1ec4462873caecda63e31b9 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 14:47:21 UTC) from {username='kudu', principal='kudu/127.0.97.62@KRBTEST.COM'} at 127.0.0.1:47260
I20260620 14:47:21.990841 1650 tablet_service.cc:1514] Processing DeleteTablet for tablet db47670da1ec4462873caecda63e31b9 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 14:47:21 UTC) from {username='kudu', principal='kudu/127.0.97.62@KRBTEST.COM'} at 127.0.0.1:56206
I20260620 14:47:21.993388 1513 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:56570:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260620 14:47:21.994117 1513 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260620 14:47:22.003823 1649 tablet_service.cc:1467] Processing CreateTablet for tablet 4488471c33004c6aa721be98c24846d9 (DEFAULT_TABLE table=test-table [id=c1ed5eedb5294050aa692f14f371e214]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 14:47:22.004525 1649 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4488471c33004c6aa721be98c24846d9. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 14:47:22.006604 2042 tablet_replica.cc:330] T db47670da1ec4462873caecda63e31b9 P 58089d127abc4bc9b8e27348383b182d: stopping tablet replica
I20260620 14:47:22.007277 2042 raft_consensus.cc:2229] T db47670da1ec4462873caecda63e31b9 P 58089d127abc4bc9b8e27348383b182d [term 1 LEADER]: Raft consensus shutting down.
I20260620 14:47:22.008139 2042 raft_consensus.cc:2258] T db47670da1ec4462873caecda63e31b9 P 58089d127abc4bc9b8e27348383b182d [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 14:47:22.008584 1928 tablet_service.cc:1467] Processing CreateTablet for tablet 4488471c33004c6aa721be98c24846d9 (DEFAULT_TABLE table=test-table [id=c1ed5eedb5294050aa692f14f371e214]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 14:47:22.009087 2043 tablet_replica.cc:330] T db47670da1ec4462873caecda63e31b9 P 796b41dc7b034f68a91a167e83106695: stopping tablet replica
I20260620 14:47:22.009179 1928 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4488471c33004c6aa721be98c24846d9. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 14:47:22.009675 2043 raft_consensus.cc:2229] T db47670da1ec4462873caecda63e31b9 P 796b41dc7b034f68a91a167e83106695 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 14:47:22.010435 1790 tablet_service.cc:1467] Processing CreateTablet for tablet 4488471c33004c6aa721be98c24846d9 (DEFAULT_TABLE table=test-table [id=c1ed5eedb5294050aa692f14f371e214]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 14:47:22.010860 1790 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4488471c33004c6aa721be98c24846d9. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 14:47:22.011143 2042 ts_tablet_manager.cc:1905] T db47670da1ec4462873caecda63e31b9 P 58089d127abc4bc9b8e27348383b182d: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 14:47:22.015515 2019 tablet_bootstrap.cc:492] T 4488471c33004c6aa721be98c24846d9 P 796b41dc7b034f68a91a167e83106695: Bootstrap starting.
I20260620 14:47:22.016009 2020 tablet_bootstrap.cc:492] T 4488471c33004c6aa721be98c24846d9 P ed946240f9ce40fbb23487b38915e4e5: Bootstrap starting.
I20260620 14:47:22.017308 2019 tablet_bootstrap.cc:654] T 4488471c33004c6aa721be98c24846d9 P 796b41dc7b034f68a91a167e83106695: Neither blocks nor log segments found. Creating new log.
I20260620 14:47:22.017580 2020 tablet_bootstrap.cc:654] T 4488471c33004c6aa721be98c24846d9 P ed946240f9ce40fbb23487b38915e4e5: Neither blocks nor log segments found. Creating new log.
I20260620 14:47:22.020074 2018 tablet_bootstrap.cc:492] T 4488471c33004c6aa721be98c24846d9 P 58089d127abc4bc9b8e27348383b182d: Bootstrap starting.
I20260620 14:47:22.020162 2043 raft_consensus.cc:2258] T db47670da1ec4462873caecda63e31b9 P 796b41dc7b034f68a91a167e83106695 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 14:47:22.022044 2018 tablet_bootstrap.cc:654] T 4488471c33004c6aa721be98c24846d9 P 58089d127abc4bc9b8e27348383b182d: Neither blocks nor log segments found. Creating new log.
I20260620 14:47:22.024767 2043 ts_tablet_manager.cc:1905] T db47670da1ec4462873caecda63e31b9 P 796b41dc7b034f68a91a167e83106695: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 14:47:22.027685 2019 tablet_bootstrap.cc:492] T 4488471c33004c6aa721be98c24846d9 P 796b41dc7b034f68a91a167e83106695: No bootstrap required, opened a new log
I20260620 14:47:22.027848 2019 ts_tablet_manager.cc:1397] T 4488471c33004c6aa721be98c24846d9 P 796b41dc7b034f68a91a167e83106695: Time spent bootstrapping tablet: real 0.012s user 0.001s sys 0.007s
I20260620 14:47:22.028812 2019 raft_consensus.cc:348] T 4488471c33004c6aa721be98c24846d9 P 796b41dc7b034f68a91a167e83106695 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "796b41dc7b034f68a91a167e83106695" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 45961 } } peers { permanent_uuid: "58089d127abc4bc9b8e27348383b182d" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 45941 } } peers { permanent_uuid: "ed946240f9ce40fbb23487b38915e4e5" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 45431 } }
I20260620 14:47:22.029050 2019 raft_consensus.cc:374] T 4488471c33004c6aa721be98c24846d9 P 796b41dc7b034f68a91a167e83106695 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 14:47:22.029152 2019 raft_consensus.cc:729] T 4488471c33004c6aa721be98c24846d9 P 796b41dc7b034f68a91a167e83106695 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 796b41dc7b034f68a91a167e83106695, State: Initialized, Role: FOLLOWER
I20260620 14:47:22.029495 2019 consensus_queue.cc:260] T 4488471c33004c6aa721be98c24846d9 P 796b41dc7b034f68a91a167e83106695 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "796b41dc7b034f68a91a167e83106695" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 45961 } } peers { permanent_uuid: "58089d127abc4bc9b8e27348383b182d" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 45941 } } peers { permanent_uuid: "ed946240f9ce40fbb23487b38915e4e5" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 45431 } }
I20260620 14:47:22.030043 2019 ts_tablet_manager.cc:1428] T 4488471c33004c6aa721be98c24846d9 P 796b41dc7b034f68a91a167e83106695: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20260620 14:47:22.032670 2020 tablet_bootstrap.cc:492] T 4488471c33004c6aa721be98c24846d9 P ed946240f9ce40fbb23487b38915e4e5: No bootstrap required, opened a new log
I20260620 14:47:22.032840 2020 ts_tablet_manager.cc:1397] T 4488471c33004c6aa721be98c24846d9 P ed946240f9ce40fbb23487b38915e4e5: Time spent bootstrapping tablet: real 0.017s user 0.005s sys 0.004s
I20260620 14:47:22.032984 2018 tablet_bootstrap.cc:492] T 4488471c33004c6aa721be98c24846d9 P 58089d127abc4bc9b8e27348383b182d: No bootstrap required, opened a new log
I20260620 14:47:22.033161 2018 ts_tablet_manager.cc:1397] T 4488471c33004c6aa721be98c24846d9 P 58089d127abc4bc9b8e27348383b182d: Time spent bootstrapping tablet: real 0.013s user 0.003s sys 0.005s
I20260620 14:47:22.033733 2020 raft_consensus.cc:348] T 4488471c33004c6aa721be98c24846d9 P ed946240f9ce40fbb23487b38915e4e5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "796b41dc7b034f68a91a167e83106695" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 45961 } } peers { permanent_uuid: "58089d127abc4bc9b8e27348383b182d" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 45941 } } peers { permanent_uuid: "ed946240f9ce40fbb23487b38915e4e5" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 45431 } }
I20260620 14:47:22.033960 2020 raft_consensus.cc:374] T 4488471c33004c6aa721be98c24846d9 P ed946240f9ce40fbb23487b38915e4e5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 14:47:22.034077 2020 raft_consensus.cc:729] T 4488471c33004c6aa721be98c24846d9 P ed946240f9ce40fbb23487b38915e4e5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ed946240f9ce40fbb23487b38915e4e5, State: Initialized, Role: FOLLOWER
I20260620 14:47:22.034184 2018 raft_consensus.cc:348] T 4488471c33004c6aa721be98c24846d9 P 58089d127abc4bc9b8e27348383b182d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "796b41dc7b034f68a91a167e83106695" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 45961 } } peers { permanent_uuid: "58089d127abc4bc9b8e27348383b182d" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 45941 } } peers { permanent_uuid: "ed946240f9ce40fbb23487b38915e4e5" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 45431 } }
I20260620 14:47:22.034471 2018 raft_consensus.cc:374] T 4488471c33004c6aa721be98c24846d9 P 58089d127abc4bc9b8e27348383b182d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 14:47:22.034359 2020 consensus_queue.cc:260] T 4488471c33004c6aa721be98c24846d9 P ed946240f9ce40fbb23487b38915e4e5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "796b41dc7b034f68a91a167e83106695" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 45961 } } peers { permanent_uuid: "58089d127abc4bc9b8e27348383b182d" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 45941 } } peers { permanent_uuid: "ed946240f9ce40fbb23487b38915e4e5" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 45431 } }
I20260620 14:47:22.034574 2018 raft_consensus.cc:729] T 4488471c33004c6aa721be98c24846d9 P 58089d127abc4bc9b8e27348383b182d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 58089d127abc4bc9b8e27348383b182d, State: Initialized, Role: FOLLOWER
I20260620 14:47:22.034931 2020 ts_tablet_manager.cc:1428] T 4488471c33004c6aa721be98c24846d9 P ed946240f9ce40fbb23487b38915e4e5: Time spent starting tablet: real 0.002s user 0.000s sys 0.000s
I20260620 14:47:22.034873 2018 consensus_queue.cc:260] T 4488471c33004c6aa721be98c24846d9 P 58089d127abc4bc9b8e27348383b182d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "796b41dc7b034f68a91a167e83106695" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 45961 } } peers { permanent_uuid: "58089d127abc4bc9b8e27348383b182d" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 45941 } } peers { permanent_uuid: "ed946240f9ce40fbb23487b38915e4e5" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 45431 } }
I20260620 14:47:22.035519 2018 ts_tablet_manager.cc:1428] T 4488471c33004c6aa721be98c24846d9 P 58089d127abc4bc9b8e27348383b182d: Time spent starting tablet: real 0.002s user 0.002s sys 0.001s
I20260620 14:47:22.038997 2042 ts_tablet_manager.cc:1918] T db47670da1ec4462873caecda63e31b9 P 58089d127abc4bc9b8e27348383b182d: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260620 14:47:22.039220 2042 log.cc:1196] T db47670da1ec4462873caecda63e31b9 P 58089d127abc4bc9b8e27348383b182d: Deleting WAL directory at /tmp/dist-test-taskUci3DU/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781966829091300-388-0/minicluster-data/ts-1/wal/wals/db47670da1ec4462873caecda63e31b9
I20260620 14:47:22.040230 2042 ts_tablet_manager.cc:1939] T db47670da1ec4462873caecda63e31b9 P 58089d127abc4bc9b8e27348383b182d: Deleting consensus metadata
I20260620 14:47:22.040364 2043 ts_tablet_manager.cc:1918] T db47670da1ec4462873caecda63e31b9 P 796b41dc7b034f68a91a167e83106695: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260620 14:47:22.040522 2043 log.cc:1196] T db47670da1ec4462873caecda63e31b9 P 796b41dc7b034f68a91a167e83106695: Deleting WAL directory at /tmp/dist-test-taskUci3DU/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781966829091300-388-0/minicluster-data/ts-0/wal/wals/db47670da1ec4462873caecda63e31b9
I20260620 14:47:22.041056 2043 ts_tablet_manager.cc:1939] T db47670da1ec4462873caecda63e31b9 P 796b41dc7b034f68a91a167e83106695: Deleting consensus metadata
I20260620 14:47:22.042912 1499 catalog_manager.cc:4841] TS 796b41dc7b034f68a91a167e83106695 (127.0.97.1:45961): tablet db47670da1ec4462873caecda63e31b9 (table test-table [id=fad12891c23e4d8fbe518ad05366785c]) successfully deleted
I20260620 14:47:22.043358 2041 tablet_replica.cc:330] T db47670da1ec4462873caecda63e31b9 P ed946240f9ce40fbb23487b38915e4e5: stopping tablet replica
I20260620 14:47:22.043889 2041 raft_consensus.cc:2229] T db47670da1ec4462873caecda63e31b9 P ed946240f9ce40fbb23487b38915e4e5 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 14:47:22.043880 1500 catalog_manager.cc:4841] TS 58089d127abc4bc9b8e27348383b182d (127.0.97.2:45941): tablet db47670da1ec4462873caecda63e31b9 (table test-table [id=fad12891c23e4d8fbe518ad05366785c]) successfully deleted
I20260620 14:47:22.044341 2041 raft_consensus.cc:2258] T db47670da1ec4462873caecda63e31b9 P ed946240f9ce40fbb23487b38915e4e5 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 14:47:22.045984 2041 ts_tablet_manager.cc:1905] T db47670da1ec4462873caecda63e31b9 P ed946240f9ce40fbb23487b38915e4e5: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 14:47:22.051067 2041 ts_tablet_manager.cc:1918] T db47670da1ec4462873caecda63e31b9 P ed946240f9ce40fbb23487b38915e4e5: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260620 14:47:22.051266 2041 log.cc:1196] T db47670da1ec4462873caecda63e31b9 P ed946240f9ce40fbb23487b38915e4e5: Deleting WAL directory at /tmp/dist-test-taskUci3DU/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781966829091300-388-0/minicluster-data/ts-2/wal/wals/db47670da1ec4462873caecda63e31b9
I20260620 14:47:22.051802 2041 ts_tablet_manager.cc:1939] T db47670da1ec4462873caecda63e31b9 P ed946240f9ce40fbb23487b38915e4e5: Deleting consensus metadata
I20260620 14:47:22.053351 1497 catalog_manager.cc:4841] TS ed946240f9ce40fbb23487b38915e4e5 (127.0.97.3:45431): tablet db47670da1ec4462873caecda63e31b9 (table test-table [id=fad12891c23e4d8fbe518ad05366785c]) successfully deleted
I20260620 14:47:22.215462 2024 raft_consensus.cc:482] T 4488471c33004c6aa721be98c24846d9 P ed946240f9ce40fbb23487b38915e4e5 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260620 14:47:22.215745 2024 raft_consensus.cc:504] T 4488471c33004c6aa721be98c24846d9 P ed946240f9ce40fbb23487b38915e4e5 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "796b41dc7b034f68a91a167e83106695" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 45961 } } peers { permanent_uuid: "58089d127abc4bc9b8e27348383b182d" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 45941 } } peers { permanent_uuid: "ed946240f9ce40fbb23487b38915e4e5" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 45431 } }
I20260620 14:47:22.217178 2024 leader_election.cc:288] T 4488471c33004c6aa721be98c24846d9 P ed946240f9ce40fbb23487b38915e4e5 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 796b41dc7b034f68a91a167e83106695 (127.0.97.1:45961), 58089d127abc4bc9b8e27348383b182d (127.0.97.2:45941)
I20260620 14:47:22.225080 2017 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:47:22.218136 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.3:56837 (local address 127.0.97.1:45961)
0620 14:47:22.218319 (+ 183us) server_negotiation.cc:206] Beginning negotiation
0620 14:47:22.218327 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0620 14:47:22.218979 (+ 652us) server_negotiation.cc:407] Connection header received
0620 14:47:22.219126 (+ 147us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:47:22.219132 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:47:22.219230 (+ 98us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:47:22.219329 (+ 99us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 14:47:22.219925 (+ 596us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:47:22.220968 (+ 1043us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:47:22.222932 (+ 1964us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:47:22.224022 (+ 1090us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:47:22.224160 (+ 138us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:47:22.224265 (+ 105us) server_negotiation.cc:1035] Waiting for connection context
0620 14:47:22.224625 (+ 360us) server_negotiation.cc:299] Negotiation successful
0620 14:47:22.224769 (+ 144us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":64}
I20260620 14:47:22.225114 2047 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:47:22.217456 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:45961 (local address 127.0.97.3:56837)
0620 14:47:22.218850 (+ 1394us) negotiation.cc:107] Waiting for socket to connect
0620 14:47:22.218867 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0620 14:47:22.218994 (+ 127us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 14:47:22.219369 (+ 375us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 14:47:22.219373 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 14:47:22.219399 (+ 26us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 14:47:22.219755 (+ 356us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:47:22.219766 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:47:22.221109 (+ 1343us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:47:22.221115 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:47:22.222762 (+ 1647us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:47:22.222775 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:47:22.224211 (+ 1436us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:47:22.224218 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:47:22.224436 (+ 218us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:47:22.224473 (+ 37us) client_negotiation.cc:769] Sending connection context
0620 14:47:22.224586 (+ 113us) client_negotiation.cc:240] Negotiation successful
0620 14:47:22.224743 (+ 157us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1295,"thread_start_us":885,"threads_started":1}
I20260620 14:47:22.226112 2013 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:47:22.217560 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.3:41005 (local address 127.0.97.2:45941)
0620 14:47:22.217906 (+ 346us) server_negotiation.cc:206] Beginning negotiation
0620 14:47:22.217913 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0620 14:47:22.218272 (+ 359us) server_negotiation.cc:407] Connection header received
0620 14:47:22.218605 (+ 333us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:47:22.218614 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:47:22.218724 (+ 110us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:47:22.218812 (+ 88us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 14:47:22.219512 (+ 700us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:47:22.220825 (+ 1313us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:47:22.223302 (+ 2477us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:47:22.224826 (+ 1524us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:47:22.224974 (+ 148us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:47:22.225061 (+ 87us) server_negotiation.cc:1035] Waiting for connection context
0620 14:47:22.225650 (+ 589us) server_negotiation.cc:299] Negotiation successful
0620 14:47:22.225752 (+ 102us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":213}
I20260620 14:47:22.226119 1670 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4488471c33004c6aa721be98c24846d9" candidate_uuid: "ed946240f9ce40fbb23487b38915e4e5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "796b41dc7b034f68a91a167e83106695" is_pre_election: true
I20260620 14:47:22.226372 1670 raft_consensus.cc:2454] T 4488471c33004c6aa721be98c24846d9 P 796b41dc7b034f68a91a167e83106695 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ed946240f9ce40fbb23487b38915e4e5 in term 0.
I20260620 14:47:22.226661 2048 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:47:22.217436 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.2:45941 (local address 127.0.97.3:41005)
0620 14:47:22.218141 (+ 705us) negotiation.cc:107] Waiting for socket to connect
0620 14:47:22.218166 (+ 25us) client_negotiation.cc:174] Beginning negotiation
0620 14:47:22.218409 (+ 243us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 14:47:22.219055 (+ 646us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 14:47:22.219061 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 14:47:22.219091 (+ 30us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 14:47:22.219295 (+ 204us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:47:22.219308 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:47:22.220991 (+ 1683us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:47:22.220997 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:47:22.223118 (+ 2121us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:47:22.223134 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:47:22.225244 (+ 2110us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:47:22.225254 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:47:22.225450 (+ 196us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:47:22.225487 (+ 37us) client_negotiation.cc:769] Sending connection context
0620 14:47:22.226139 (+ 652us) client_negotiation.cc:240] Negotiation successful
0620 14:47:22.226299 (+ 160us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":559,"mutex_wait_us":2,"thread_start_us":1201,"threads_started":1}
I20260620 14:47:22.226915 1882 leader_election.cc:302] T 4488471c33004c6aa721be98c24846d9 P ed946240f9ce40fbb23487b38915e4e5 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 796b41dc7b034f68a91a167e83106695, ed946240f9ce40fbb23487b38915e4e5; no voters:
I20260620 14:47:22.227367 2024 raft_consensus.cc:2781] T 4488471c33004c6aa721be98c24846d9 P ed946240f9ce40fbb23487b38915e4e5 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260620 14:47:22.227491 2024 raft_consensus.cc:482] T 4488471c33004c6aa721be98c24846d9 P ed946240f9ce40fbb23487b38915e4e5 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260620 14:47:22.227588 2024 raft_consensus.cc:3037] T 4488471c33004c6aa721be98c24846d9 P ed946240f9ce40fbb23487b38915e4e5 [term 0 FOLLOWER]: Advancing to term 1
I20260620 14:47:22.227626 1810 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4488471c33004c6aa721be98c24846d9" candidate_uuid: "ed946240f9ce40fbb23487b38915e4e5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "58089d127abc4bc9b8e27348383b182d" is_pre_election: true
I20260620 14:47:22.228117 1810 raft_consensus.cc:2454] T 4488471c33004c6aa721be98c24846d9 P 58089d127abc4bc9b8e27348383b182d [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ed946240f9ce40fbb23487b38915e4e5 in term 0.
I20260620 14:47:22.231283 2024 raft_consensus.cc:504] T 4488471c33004c6aa721be98c24846d9 P ed946240f9ce40fbb23487b38915e4e5 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "796b41dc7b034f68a91a167e83106695" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 45961 } } peers { permanent_uuid: "58089d127abc4bc9b8e27348383b182d" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 45941 } } peers { permanent_uuid: "ed946240f9ce40fbb23487b38915e4e5" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 45431 } }
I20260620 14:47:22.232000 2024 leader_election.cc:288] T 4488471c33004c6aa721be98c24846d9 P ed946240f9ce40fbb23487b38915e4e5 [CANDIDATE]: Term 1 election: Requested vote from peers 796b41dc7b034f68a91a167e83106695 (127.0.97.1:45961), 58089d127abc4bc9b8e27348383b182d (127.0.97.2:45941)
I20260620 14:47:22.232511 1810 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4488471c33004c6aa721be98c24846d9" candidate_uuid: "ed946240f9ce40fbb23487b38915e4e5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "58089d127abc4bc9b8e27348383b182d"
I20260620 14:47:22.232548 1670 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4488471c33004c6aa721be98c24846d9" candidate_uuid: "ed946240f9ce40fbb23487b38915e4e5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "796b41dc7b034f68a91a167e83106695"
I20260620 14:47:22.232738 1810 raft_consensus.cc:3037] T 4488471c33004c6aa721be98c24846d9 P 58089d127abc4bc9b8e27348383b182d [term 0 FOLLOWER]: Advancing to term 1
I20260620 14:47:22.232769 1670 raft_consensus.cc:3037] T 4488471c33004c6aa721be98c24846d9 P 796b41dc7b034f68a91a167e83106695 [term 0 FOLLOWER]: Advancing to term 1
I20260620 14:47:22.235289 1810 raft_consensus.cc:2454] T 4488471c33004c6aa721be98c24846d9 P 58089d127abc4bc9b8e27348383b182d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ed946240f9ce40fbb23487b38915e4e5 in term 1.
I20260620 14:47:22.235774 1670 raft_consensus.cc:2454] T 4488471c33004c6aa721be98c24846d9 P 796b41dc7b034f68a91a167e83106695 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ed946240f9ce40fbb23487b38915e4e5 in term 1.
I20260620 14:47:22.235821 1883 leader_election.cc:302] T 4488471c33004c6aa721be98c24846d9 P ed946240f9ce40fbb23487b38915e4e5 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 58089d127abc4bc9b8e27348383b182d, ed946240f9ce40fbb23487b38915e4e5; no voters:
I20260620 14:47:22.236198 2024 raft_consensus.cc:2781] T 4488471c33004c6aa721be98c24846d9 P ed946240f9ce40fbb23487b38915e4e5 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 14:47:22.236687 2024 raft_consensus.cc:686] T 4488471c33004c6aa721be98c24846d9 P ed946240f9ce40fbb23487b38915e4e5 [term 1 LEADER]: Becoming Leader. State: Replica: ed946240f9ce40fbb23487b38915e4e5, State: Running, Role: LEADER
I20260620 14:47:22.237349 2024 consensus_queue.cc:237] T 4488471c33004c6aa721be98c24846d9 P ed946240f9ce40fbb23487b38915e4e5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "796b41dc7b034f68a91a167e83106695" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 45961 } } peers { permanent_uuid: "58089d127abc4bc9b8e27348383b182d" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 45941 } } peers { permanent_uuid: "ed946240f9ce40fbb23487b38915e4e5" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 45431 } }
I20260620 14:47:22.241415 1511 catalog_manager.cc:5495] T 4488471c33004c6aa721be98c24846d9 P ed946240f9ce40fbb23487b38915e4e5 reported cstate change: term changed from 0 to 1, leader changed from <none> to ed946240f9ce40fbb23487b38915e4e5 (127.0.97.3). New cstate: current_term: 1 leader_uuid: "ed946240f9ce40fbb23487b38915e4e5" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "796b41dc7b034f68a91a167e83106695" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 45961 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "58089d127abc4bc9b8e27348383b182d" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 45941 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "ed946240f9ce40fbb23487b38915e4e5" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 45431 } health_report { overall_health: HEALTHY } } }
Jun 20 14:47:22 dist-test-slave-q9j6 krb5kdc[1468](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781966839, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.62@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260620 14:47:22.302054 2057 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:47:22.285961 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:37781 (local address 127.0.0.1:56580)
0620 14:47:22.286907 (+ 946us) negotiation.cc:107] Waiting for socket to connect
0620 14:47:22.286923 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0620 14:47:22.287052 (+ 129us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 14:47:22.287424 (+ 372us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 14:47:22.287427 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 14:47:22.288554 (+ 1127us) client_negotiation.cc:189] Negotiated authn=SASL
0620 14:47:22.288774 (+ 220us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:47:22.288784 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:47:22.290291 (+ 1507us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:47:22.290295 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:47:22.291201 (+ 906us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:47:22.291211 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:47:22.292222 (+ 1011us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:47:22.292226 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:47:22.292400 (+ 174us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:47:22.293117 (+ 717us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 14:47:22.293134 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0620 14:47:22.295269 (+ 2135us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 14:47:22.299677 (+ 4408us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 14:47:22.299688 (+ 11us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 14:47:22.299695 (+ 7us) client_negotiation.cc:760] calling sasl_client_step()
0620 14:47:22.300109 (+ 414us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 14:47:22.300533 (+ 424us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 14:47:22.300536 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 14:47:22.300538 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 14:47:22.300666 (+ 128us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 14:47:22.301295 (+ 629us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 14:47:22.301301 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 14:47:22.301437 (+ 136us) client_negotiation.cc:769] Sending connection context
0620 14:47:22.301569 (+ 132us) client_negotiation.cc:240] Negotiation successful
0620 14:47:22.301774 (+ 205us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":798,"thread_start_us":642,"threads_started":1}
I20260620 14:47:22.302387 2056 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:47:22.285975 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56580 (local address 127.0.97.62:37781)
0620 14:47:22.286691 (+ 716us) server_negotiation.cc:206] Beginning negotiation
0620 14:47:22.286699 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0620 14:47:22.287028 (+ 329us) server_negotiation.cc:407] Connection header received
0620 14:47:22.287168 (+ 140us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:47:22.287174 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:47:22.287262 (+ 88us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:47:22.287394 (+ 132us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:47:22.288984 (+ 1590us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:47:22.290129 (+ 1145us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:47:22.291354 (+ 1225us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:47:22.292057 (+ 703us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:47:22.292207 (+ 150us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:47:22.295476 (+ 3269us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 14:47:22.295483 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 14:47:22.295488 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 14:47:22.295524 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0620 14:47:22.299492 (+ 3968us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:47:22.300248 (+ 756us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:47:22.300253 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:47:22.300257 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:47:22.300418 (+ 161us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:47:22.300779 (+ 361us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:47:22.300784 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:47:22.300787 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:47:22.301151 (+ 364us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 14:47:22.301319 (+ 168us) server_negotiation.cc:1035] Waiting for connection context
0620 14:47:22.301697 (+ 378us) server_negotiation.cc:299] Negotiation successful
0620 14:47:22.301928 (+ 231us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":17,"server-negotiator.queue_time_us":544,"thread_start_us":381,"threads_started":1}
I20260620 14:47:22.307564 1511 catalog_manager.cc:1908] Servicing CreateTable request from {username='kudu', principal='kudu/127.0.97.62@KRBTEST.COM'} at 127.0.0.1:56580:
name: "kudu_system.kudu_transactions"
schema {
columns {
name: "txn_id"
type: INT64
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "entry_type"
type: INT8
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "identifier"
type: STRING
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "metadata"
type: STRING
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
rows: "<redacted>""\006\001\000\000\000\000\000\000\000\000\007\001@B\017\000\000\000\000\000"
indirect_data: "<redacted>"""
}
partition_schema {
range_schema {
columns {
name: "txn_id"
}
}
}
table_type: TXN_STATUS_TABLE
W20260620 14:47:22.308852 1511 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table kudu_system.kudu_transactions in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260620 14:47:22.315972 1790 tablet_service.cc:1467] Processing CreateTablet for tablet e430b49db78d4a81832c542200c08980 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=1bca8f41d8254190a25b237780f4112c]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260620 14:47:22.316074 1649 tablet_service.cc:1467] Processing CreateTablet for tablet e430b49db78d4a81832c542200c08980 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=1bca8f41d8254190a25b237780f4112c]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260620 14:47:22.316550 1790 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e430b49db78d4a81832c542200c08980. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 14:47:22.316610 1649 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e430b49db78d4a81832c542200c08980. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 14:47:22.322456 2018 tablet_bootstrap.cc:492] T e430b49db78d4a81832c542200c08980 P 58089d127abc4bc9b8e27348383b182d: Bootstrap starting.
I20260620 14:47:22.324218 2018 tablet_bootstrap.cc:654] T e430b49db78d4a81832c542200c08980 P 58089d127abc4bc9b8e27348383b182d: Neither blocks nor log segments found. Creating new log.
I20260620 14:47:22.324397 1929 tablet_service.cc:1467] Processing CreateTablet for tablet e430b49db78d4a81832c542200c08980 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=1bca8f41d8254190a25b237780f4112c]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260620 14:47:22.324719 2019 tablet_bootstrap.cc:492] T e430b49db78d4a81832c542200c08980 P 796b41dc7b034f68a91a167e83106695: Bootstrap starting.
I20260620 14:47:22.324939 1929 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e430b49db78d4a81832c542200c08980. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 14:47:22.326807 2019 tablet_bootstrap.cc:654] T e430b49db78d4a81832c542200c08980 P 796b41dc7b034f68a91a167e83106695: Neither blocks nor log segments found. Creating new log.
I20260620 14:47:22.334403 2020 tablet_bootstrap.cc:492] T e430b49db78d4a81832c542200c08980 P ed946240f9ce40fbb23487b38915e4e5: Bootstrap starting.
I20260620 14:47:22.336206 2020 tablet_bootstrap.cc:654] T e430b49db78d4a81832c542200c08980 P ed946240f9ce40fbb23487b38915e4e5: Neither blocks nor log segments found. Creating new log.
I20260620 14:47:22.337005 2018 tablet_bootstrap.cc:492] T e430b49db78d4a81832c542200c08980 P 58089d127abc4bc9b8e27348383b182d: No bootstrap required, opened a new log
I20260620 14:47:22.337240 2018 ts_tablet_manager.cc:1397] T e430b49db78d4a81832c542200c08980 P 58089d127abc4bc9b8e27348383b182d: Time spent bootstrapping tablet: real 0.015s user 0.010s sys 0.000s
I20260620 14:47:22.339838 2019 tablet_bootstrap.cc:492] T e430b49db78d4a81832c542200c08980 P 796b41dc7b034f68a91a167e83106695: No bootstrap required, opened a new log
I20260620 14:47:22.340008 2019 ts_tablet_manager.cc:1397] T e430b49db78d4a81832c542200c08980 P 796b41dc7b034f68a91a167e83106695: Time spent bootstrapping tablet: real 0.015s user 0.002s sys 0.010s
I20260620 14:47:22.339823 2018 raft_consensus.cc:348] T e430b49db78d4a81832c542200c08980 P 58089d127abc4bc9b8e27348383b182d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "796b41dc7b034f68a91a167e83106695" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 45961 } } peers { permanent_uuid: "58089d127abc4bc9b8e27348383b182d" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 45941 } } peers { permanent_uuid: "ed946240f9ce40fbb23487b38915e4e5" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 45431 } }
I20260620 14:47:22.340161 2018 raft_consensus.cc:374] T e430b49db78d4a81832c542200c08980 P 58089d127abc4bc9b8e27348383b182d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 14:47:22.340261 2018 raft_consensus.cc:729] T e430b49db78d4a81832c542200c08980 P 58089d127abc4bc9b8e27348383b182d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 58089d127abc4bc9b8e27348383b182d, State: Initialized, Role: FOLLOWER
I20260620 14:47:22.340583 2018 consensus_queue.cc:260] T e430b49db78d4a81832c542200c08980 P 58089d127abc4bc9b8e27348383b182d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "796b41dc7b034f68a91a167e83106695" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 45961 } } peers { permanent_uuid: "58089d127abc4bc9b8e27348383b182d" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 45941 } } peers { permanent_uuid: "ed946240f9ce40fbb23487b38915e4e5" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 45431 } }
I20260620 14:47:22.340777 2019 raft_consensus.cc:348] T e430b49db78d4a81832c542200c08980 P 796b41dc7b034f68a91a167e83106695 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "796b41dc7b034f68a91a167e83106695" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 45961 } } peers { permanent_uuid: "58089d127abc4bc9b8e27348383b182d" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 45941 } } peers { permanent_uuid: "ed946240f9ce40fbb23487b38915e4e5" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 45431 } }
I20260620 14:47:22.341208 2025 tablet_replica.cc:439] T e430b49db78d4a81832c542200c08980 P 58089d127abc4bc9b8e27348383b182d: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "796b41dc7b034f68a91a167e83106695" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 45961 } } peers { permanent_uuid: "58089d127abc4bc9b8e27348383b182d" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 45941 } } peers { permanent_uuid: "ed946240f9ce40fbb23487b38915e4e5" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 45431 } } }
I20260620 14:47:22.341514 2025 tablet_replica.cc:442] T e430b49db78d4a81832c542200c08980 P 58089d127abc4bc9b8e27348383b182d: This TxnStatusTablet replica's current role is: FOLLOWER
I20260620 14:47:22.342291 2019 raft_consensus.cc:374] T e430b49db78d4a81832c542200c08980 P 796b41dc7b034f68a91a167e83106695 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 14:47:22.342391 2019 raft_consensus.cc:729] T e430b49db78d4a81832c542200c08980 P 796b41dc7b034f68a91a167e83106695 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 796b41dc7b034f68a91a167e83106695, State: Initialized, Role: FOLLOWER
I20260620 14:47:22.342790 2018 ts_tablet_manager.cc:1428] T e430b49db78d4a81832c542200c08980 P 58089d127abc4bc9b8e27348383b182d: Time spent starting tablet: real 0.004s user 0.002s sys 0.000s
I20260620 14:47:22.342646 2019 consensus_queue.cc:260] T e430b49db78d4a81832c542200c08980 P 796b41dc7b034f68a91a167e83106695 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "796b41dc7b034f68a91a167e83106695" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 45961 } } peers { permanent_uuid: "58089d127abc4bc9b8e27348383b182d" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 45941 } } peers { permanent_uuid: "ed946240f9ce40fbb23487b38915e4e5" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 45431 } }
I20260620 14:47:22.343127 2026 tablet_replica.cc:439] T e430b49db78d4a81832c542200c08980 P 796b41dc7b034f68a91a167e83106695: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "796b41dc7b034f68a91a167e83106695" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 45961 } } peers { permanent_uuid: "58089d127abc4bc9b8e27348383b182d" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 45941 } } peers { permanent_uuid: "ed946240f9ce40fbb23487b38915e4e5" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 45431 } } }
I20260620 14:47:22.343526 2026 tablet_replica.cc:442] T e430b49db78d4a81832c542200c08980 P 796b41dc7b034f68a91a167e83106695: This TxnStatusTablet replica's current role is: FOLLOWER
I20260620 14:47:22.344162 2019 ts_tablet_manager.cc:1428] T e430b49db78d4a81832c542200c08980 P 796b41dc7b034f68a91a167e83106695: Time spent starting tablet: real 0.004s user 0.002s sys 0.000s
I20260620 14:47:22.351912 2020 tablet_bootstrap.cc:492] T e430b49db78d4a81832c542200c08980 P ed946240f9ce40fbb23487b38915e4e5: No bootstrap required, opened a new log
I20260620 14:47:22.352115 2020 ts_tablet_manager.cc:1397] T e430b49db78d4a81832c542200c08980 P ed946240f9ce40fbb23487b38915e4e5: Time spent bootstrapping tablet: real 0.018s user 0.008s sys 0.004s
I20260620 14:47:22.352732 2020 raft_consensus.cc:348] T e430b49db78d4a81832c542200c08980 P ed946240f9ce40fbb23487b38915e4e5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "796b41dc7b034f68a91a167e83106695" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 45961 } } peers { permanent_uuid: "58089d127abc4bc9b8e27348383b182d" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 45941 } } peers { permanent_uuid: "ed946240f9ce40fbb23487b38915e4e5" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 45431 } }
I20260620 14:47:22.352967 2020 raft_consensus.cc:374] T e430b49db78d4a81832c542200c08980 P ed946240f9ce40fbb23487b38915e4e5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 14:47:22.353063 2020 raft_consensus.cc:729] T e430b49db78d4a81832c542200c08980 P ed946240f9ce40fbb23487b38915e4e5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ed946240f9ce40fbb23487b38915e4e5, State: Initialized, Role: FOLLOWER
I20260620 14:47:22.353317 2020 consensus_queue.cc:260] T e430b49db78d4a81832c542200c08980 P ed946240f9ce40fbb23487b38915e4e5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "796b41dc7b034f68a91a167e83106695" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 45961 } } peers { permanent_uuid: "58089d127abc4bc9b8e27348383b182d" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 45941 } } peers { permanent_uuid: "ed946240f9ce40fbb23487b38915e4e5" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 45431 } }
I20260620 14:47:22.353706 2024 tablet_replica.cc:439] T e430b49db78d4a81832c542200c08980 P ed946240f9ce40fbb23487b38915e4e5: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "796b41dc7b034f68a91a167e83106695" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 45961 } } peers { permanent_uuid: "58089d127abc4bc9b8e27348383b182d" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 45941 } } peers { permanent_uuid: "ed946240f9ce40fbb23487b38915e4e5" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 45431 } } }
I20260620 14:47:22.353906 2020 ts_tablet_manager.cc:1428] T e430b49db78d4a81832c542200c08980 P ed946240f9ce40fbb23487b38915e4e5: Time spent starting tablet: real 0.002s user 0.000s sys 0.000s
I20260620 14:47:22.354053 2024 tablet_replica.cc:442] T e430b49db78d4a81832c542200c08980 P ed946240f9ce40fbb23487b38915e4e5: This TxnStatusTablet replica's current role is: FOLLOWER
I20260620 14:47:22.515928 2024 raft_consensus.cc:482] T e430b49db78d4a81832c542200c08980 P ed946240f9ce40fbb23487b38915e4e5 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260620 14:47:22.516175 2024 raft_consensus.cc:504] T e430b49db78d4a81832c542200c08980 P ed946240f9ce40fbb23487b38915e4e5 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "796b41dc7b034f68a91a167e83106695" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 45961 } } peers { permanent_uuid: "58089d127abc4bc9b8e27348383b182d" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 45941 } } peers { permanent_uuid: "ed946240f9ce40fbb23487b38915e4e5" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 45431 } }
I20260620 14:47:22.516845 2024 leader_election.cc:288] T e430b49db78d4a81832c542200c08980 P ed946240f9ce40fbb23487b38915e4e5 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 796b41dc7b034f68a91a167e83106695 (127.0.97.1:45961), 58089d127abc4bc9b8e27348383b182d (127.0.97.2:45941)
I20260620 14:47:22.517446 1810 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e430b49db78d4a81832c542200c08980" candidate_uuid: "ed946240f9ce40fbb23487b38915e4e5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "58089d127abc4bc9b8e27348383b182d" is_pre_election: true
I20260620 14:47:22.517485 1670 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e430b49db78d4a81832c542200c08980" candidate_uuid: "ed946240f9ce40fbb23487b38915e4e5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "796b41dc7b034f68a91a167e83106695" is_pre_election: true
I20260620 14:47:22.517714 1810 raft_consensus.cc:2454] T e430b49db78d4a81832c542200c08980 P 58089d127abc4bc9b8e27348383b182d [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ed946240f9ce40fbb23487b38915e4e5 in term 0.
I20260620 14:47:22.517773 1670 raft_consensus.cc:2454] T e430b49db78d4a81832c542200c08980 P 796b41dc7b034f68a91a167e83106695 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ed946240f9ce40fbb23487b38915e4e5 in term 0.
I20260620 14:47:22.518188 1883 leader_election.cc:302] T e430b49db78d4a81832c542200c08980 P ed946240f9ce40fbb23487b38915e4e5 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 58089d127abc4bc9b8e27348383b182d, ed946240f9ce40fbb23487b38915e4e5; no voters:
I20260620 14:47:22.518518 2024 raft_consensus.cc:2781] T e430b49db78d4a81832c542200c08980 P ed946240f9ce40fbb23487b38915e4e5 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260620 14:47:22.518616 2024 raft_consensus.cc:482] T e430b49db78d4a81832c542200c08980 P ed946240f9ce40fbb23487b38915e4e5 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260620 14:47:22.518736 2024 raft_consensus.cc:3037] T e430b49db78d4a81832c542200c08980 P ed946240f9ce40fbb23487b38915e4e5 [term 0 FOLLOWER]: Advancing to term 1
I20260620 14:47:22.521191 2024 raft_consensus.cc:504] T e430b49db78d4a81832c542200c08980 P ed946240f9ce40fbb23487b38915e4e5 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "796b41dc7b034f68a91a167e83106695" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 45961 } } peers { permanent_uuid: "58089d127abc4bc9b8e27348383b182d" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 45941 } } peers { permanent_uuid: "ed946240f9ce40fbb23487b38915e4e5" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 45431 } }
I20260620 14:47:22.521844 2024 leader_election.cc:288] T e430b49db78d4a81832c542200c08980 P ed946240f9ce40fbb23487b38915e4e5 [CANDIDATE]: Term 1 election: Requested vote from peers 796b41dc7b034f68a91a167e83106695 (127.0.97.1:45961), 58089d127abc4bc9b8e27348383b182d (127.0.97.2:45941)
I20260620 14:47:22.522302 1670 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e430b49db78d4a81832c542200c08980" candidate_uuid: "ed946240f9ce40fbb23487b38915e4e5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "796b41dc7b034f68a91a167e83106695"
I20260620 14:47:22.522320 1810 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e430b49db78d4a81832c542200c08980" candidate_uuid: "ed946240f9ce40fbb23487b38915e4e5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "58089d127abc4bc9b8e27348383b182d"
I20260620 14:47:22.522545 1670 raft_consensus.cc:3037] T e430b49db78d4a81832c542200c08980 P 796b41dc7b034f68a91a167e83106695 [term 0 FOLLOWER]: Advancing to term 1
I20260620 14:47:22.522547 1810 raft_consensus.cc:3037] T e430b49db78d4a81832c542200c08980 P 58089d127abc4bc9b8e27348383b182d [term 0 FOLLOWER]: Advancing to term 1
I20260620 14:47:22.524818 1810 raft_consensus.cc:2454] T e430b49db78d4a81832c542200c08980 P 58089d127abc4bc9b8e27348383b182d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ed946240f9ce40fbb23487b38915e4e5 in term 1.
I20260620 14:47:22.525120 1670 raft_consensus.cc:2454] T e430b49db78d4a81832c542200c08980 P 796b41dc7b034f68a91a167e83106695 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ed946240f9ce40fbb23487b38915e4e5 in term 1.
I20260620 14:47:22.525282 1883 leader_election.cc:302] T e430b49db78d4a81832c542200c08980 P ed946240f9ce40fbb23487b38915e4e5 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 58089d127abc4bc9b8e27348383b182d, ed946240f9ce40fbb23487b38915e4e5; no voters:
I20260620 14:47:22.525585 2024 raft_consensus.cc:2781] T e430b49db78d4a81832c542200c08980 P ed946240f9ce40fbb23487b38915e4e5 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 14:47:22.525753 2024 raft_consensus.cc:686] T e430b49db78d4a81832c542200c08980 P ed946240f9ce40fbb23487b38915e4e5 [term 1 LEADER]: Becoming Leader. State: Replica: ed946240f9ce40fbb23487b38915e4e5, State: Running, Role: LEADER
I20260620 14:47:22.526015 2024 consensus_queue.cc:237] T e430b49db78d4a81832c542200c08980 P ed946240f9ce40fbb23487b38915e4e5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "796b41dc7b034f68a91a167e83106695" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 45961 } } peers { permanent_uuid: "58089d127abc4bc9b8e27348383b182d" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 45941 } } peers { permanent_uuid: "ed946240f9ce40fbb23487b38915e4e5" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 45431 } }
I20260620 14:47:22.527684 2049 tablet_replica.cc:439] T e430b49db78d4a81832c542200c08980 P ed946240f9ce40fbb23487b38915e4e5: TxnStatusTablet state changed. Reason: New leader ed946240f9ce40fbb23487b38915e4e5. Latest consensus state: current_term: 1 leader_uuid: "ed946240f9ce40fbb23487b38915e4e5" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "796b41dc7b034f68a91a167e83106695" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 45961 } } peers { permanent_uuid: "58089d127abc4bc9b8e27348383b182d" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 45941 } } peers { permanent_uuid: "ed946240f9ce40fbb23487b38915e4e5" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 45431 } } }
I20260620 14:47:22.527961 2049 tablet_replica.cc:442] T e430b49db78d4a81832c542200c08980 P ed946240f9ce40fbb23487b38915e4e5: This TxnStatusTablet replica's current role is: LEADER
I20260620 14:47:22.528473 2062 txn_status_manager.cc:873] Waiting until node catch up with all replicated operations in previous term...
I20260620 14:47:22.528709 2062 txn_status_manager.cc:929] Loading transaction status metadata into memory...
I20260620 14:47:22.529219 1511 catalog_manager.cc:5495] T e430b49db78d4a81832c542200c08980 P ed946240f9ce40fbb23487b38915e4e5 reported cstate change: term changed from 0 to 1, leader changed from <none> to ed946240f9ce40fbb23487b38915e4e5 (127.0.97.3). New cstate: current_term: 1 leader_uuid: "ed946240f9ce40fbb23487b38915e4e5" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "796b41dc7b034f68a91a167e83106695" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 45961 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "58089d127abc4bc9b8e27348383b182d" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 45941 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "ed946240f9ce40fbb23487b38915e4e5" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 45431 } health_report { overall_health: HEALTHY } } }
I20260620 14:47:22.585842 2057 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:47:22.578730 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.3:45431 (local address 127.0.0.1:41174)
0620 14:47:22.579057 (+ 327us) negotiation.cc:107] Waiting for socket to connect
0620 14:47:22.579085 (+ 28us) client_negotiation.cc:174] Beginning negotiation
0620 14:47:22.579247 (+ 162us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 14:47:22.579890 (+ 643us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 14:47:22.579894 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 14:47:22.579904 (+ 10us) client_negotiation.cc:189] Negotiated authn=TOKEN
0620 14:47:22.580111 (+ 207us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:47:22.580119 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:47:22.581660 (+ 1541us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:47:22.581664 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:47:22.582687 (+ 1023us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:47:22.582698 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:47:22.583732 (+ 1034us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:47:22.583737 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:47:22.583882 (+ 145us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:47:22.583944 (+ 62us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0620 14:47:22.585170 (+ 1226us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0620 14:47:22.585223 (+ 53us) client_negotiation.cc:769] Sending connection context
0620 14:47:22.585323 (+ 100us) client_negotiation.cc:240] Negotiation successful
0620 14:47:22.585514 (+ 191us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":68}
I20260620 14:47:22.585889 2065 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:47:22.578715 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41174 (local address 127.0.97.3:45431)
0620 14:47:22.579392 (+ 677us) server_negotiation.cc:206] Beginning negotiation
0620 14:47:22.579399 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0620 14:47:22.579420 (+ 21us) server_negotiation.cc:407] Connection header received
0620 14:47:22.579524 (+ 104us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:47:22.579533 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:47:22.579680 (+ 147us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:47:22.579853 (+ 173us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 14:47:22.580275 (+ 422us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:47:22.581515 (+ 1240us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:47:22.582840 (+ 1325us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:47:22.583536 (+ 696us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:47:22.583702 (+ 166us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:47:22.584140 (+ 438us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 14:47:22.585032 (+ 892us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0620 14:47:22.585190 (+ 158us) server_negotiation.cc:1035] Waiting for connection context
0620 14:47:22.585392 (+ 202us) server_negotiation.cc:299] Negotiation successful
0620 14:47:22.585488 (+ 96us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":521,"thread_start_us":387,"threads_started":1}
I20260620 14:47:22.591470 1670 raft_consensus.cc:1261] T e430b49db78d4a81832c542200c08980 P 796b41dc7b034f68a91a167e83106695 [term 1 FOLLOWER]: Refusing update from remote peer ed946240f9ce40fbb23487b38915e4e5: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 14:47:22.591699 1810 raft_consensus.cc:1261] T e430b49db78d4a81832c542200c08980 P 58089d127abc4bc9b8e27348383b182d [term 1 FOLLOWER]: Refusing update from remote peer ed946240f9ce40fbb23487b38915e4e5: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 14:47:22.592563 2049 consensus_queue.cc:1035] T e430b49db78d4a81832c542200c08980 P ed946240f9ce40fbb23487b38915e4e5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "796b41dc7b034f68a91a167e83106695" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 45961 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 14:47:22.592911 2024 consensus_queue.cc:1035] T e430b49db78d4a81832c542200c08980 P ed946240f9ce40fbb23487b38915e4e5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "58089d127abc4bc9b8e27348383b182d" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 45941 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 14:47:22.616118 2026 tablet_replica.cc:439] T e430b49db78d4a81832c542200c08980 P 796b41dc7b034f68a91a167e83106695: TxnStatusTablet state changed. Reason: New leader ed946240f9ce40fbb23487b38915e4e5. Latest consensus state: current_term: 1 leader_uuid: "ed946240f9ce40fbb23487b38915e4e5" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "796b41dc7b034f68a91a167e83106695" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 45961 } } peers { permanent_uuid: "58089d127abc4bc9b8e27348383b182d" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 45941 } } peers { permanent_uuid: "ed946240f9ce40fbb23487b38915e4e5" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 45431 } } }
I20260620 14:47:22.616465 2026 tablet_replica.cc:442] T e430b49db78d4a81832c542200c08980 P 796b41dc7b034f68a91a167e83106695: This TxnStatusTablet replica's current role is: FOLLOWER
I20260620 14:47:22.616761 2066 mvcc.cc:205] Tried to move back new op lower bound from 7298936187246682112 to 7298936186988621824. Current Snapshot: MvccSnapshot[applied={T|T < 7298936187246682112}]
I20260620 14:47:22.629395 2025 tablet_replica.cc:439] T e430b49db78d4a81832c542200c08980 P 58089d127abc4bc9b8e27348383b182d: TxnStatusTablet state changed. Reason: New leader ed946240f9ce40fbb23487b38915e4e5. Latest consensus state: current_term: 1 leader_uuid: "ed946240f9ce40fbb23487b38915e4e5" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "796b41dc7b034f68a91a167e83106695" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 45961 } } peers { permanent_uuid: "58089d127abc4bc9b8e27348383b182d" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 45941 } } peers { permanent_uuid: "ed946240f9ce40fbb23487b38915e4e5" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 45431 } } }
I20260620 14:47:22.629743 2025 tablet_replica.cc:442] T e430b49db78d4a81832c542200c08980 P 58089d127abc4bc9b8e27348383b182d: This TxnStatusTablet replica's current role is: FOLLOWER
I20260620 14:47:22.629680 2072 tablet_replica.cc:439] T e430b49db78d4a81832c542200c08980 P 58089d127abc4bc9b8e27348383b182d: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "ed946240f9ce40fbb23487b38915e4e5" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "796b41dc7b034f68a91a167e83106695" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 45961 } } peers { permanent_uuid: "58089d127abc4bc9b8e27348383b182d" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 45941 } } peers { permanent_uuid: "ed946240f9ce40fbb23487b38915e4e5" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 45431 } } }
I20260620 14:47:22.629995 2072 tablet_replica.cc:442] T e430b49db78d4a81832c542200c08980 P 58089d127abc4bc9b8e27348383b182d: This TxnStatusTablet replica's current role is: FOLLOWER
I20260620 14:47:22.639855 2024 tablet_replica.cc:439] T e430b49db78d4a81832c542200c08980 P ed946240f9ce40fbb23487b38915e4e5: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "ed946240f9ce40fbb23487b38915e4e5" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "796b41dc7b034f68a91a167e83106695" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 45961 } } peers { permanent_uuid: "58089d127abc4bc9b8e27348383b182d" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 45941 } } peers { permanent_uuid: "ed946240f9ce40fbb23487b38915e4e5" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 45431 } } }
I20260620 14:47:22.640201 2024 tablet_replica.cc:442] T e430b49db78d4a81832c542200c08980 P ed946240f9ce40fbb23487b38915e4e5: This TxnStatusTablet replica's current role is: LEADER
I20260620 14:47:22.640652 2026 tablet_replica.cc:439] T e430b49db78d4a81832c542200c08980 P 796b41dc7b034f68a91a167e83106695: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "ed946240f9ce40fbb23487b38915e4e5" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "796b41dc7b034f68a91a167e83106695" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 45961 } } peers { permanent_uuid: "58089d127abc4bc9b8e27348383b182d" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 45941 } } peers { permanent_uuid: "ed946240f9ce40fbb23487b38915e4e5" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 45431 } } }
I20260620 14:47:22.640913 2026 tablet_replica.cc:442] T e430b49db78d4a81832c542200c08980 P 796b41dc7b034f68a91a167e83106695: This TxnStatusTablet replica's current role is: FOLLOWER
I20260620 14:47:22.640869 2024 tablet_replica.cc:439] T e430b49db78d4a81832c542200c08980 P ed946240f9ce40fbb23487b38915e4e5: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "ed946240f9ce40fbb23487b38915e4e5" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "796b41dc7b034f68a91a167e83106695" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 45961 } } peers { permanent_uuid: "58089d127abc4bc9b8e27348383b182d" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 45941 } } peers { permanent_uuid: "ed946240f9ce40fbb23487b38915e4e5" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 45431 } } }
I20260620 14:47:22.641201 2024 tablet_replica.cc:442] T e430b49db78d4a81832c542200c08980 P ed946240f9ce40fbb23487b38915e4e5: This TxnStatusTablet replica's current role is: LEADER
I20260620 14:47:22.656383 2065 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:47:22.650682 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41178 (local address 127.0.97.3:45431)
0620 14:47:22.650894 (+ 212us) server_negotiation.cc:206] Beginning negotiation
0620 14:47:22.650903 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0620 14:47:22.651480 (+ 577us) server_negotiation.cc:407] Connection header received
0620 14:47:22.651709 (+ 229us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:47:22.651715 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:47:22.651786 (+ 71us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:47:22.651893 (+ 107us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 14:47:22.652330 (+ 437us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:47:22.653089 (+ 759us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:47:22.654458 (+ 1369us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:47:22.654893 (+ 435us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:47:22.655042 (+ 149us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:47:22.655526 (+ 484us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 14:47:22.655698 (+ 172us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0620 14:47:22.655798 (+ 100us) server_negotiation.cc:1035] Waiting for connection context
0620 14:47:22.655933 (+ 135us) server_negotiation.cc:299] Negotiation successful
0620 14:47:22.656021 (+ 88us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":73}
I20260620 14:47:22.658232 2024 consensus_queue.cc:1035] T 4488471c33004c6aa721be98c24846d9 P ed946240f9ce40fbb23487b38915e4e5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "796b41dc7b034f68a91a167e83106695" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 45961 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 14:47:22.664206 2049 consensus_queue.cc:1035] T 4488471c33004c6aa721be98c24846d9 P ed946240f9ce40fbb23487b38915e4e5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "58089d127abc4bc9b8e27348383b182d" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 45941 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 14:47:22.686558 2065 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:47:22.679480 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.3:46915 (local address 127.0.97.3:45431)
0620 14:47:22.679706 (+ 226us) server_negotiation.cc:206] Beginning negotiation
0620 14:47:22.679712 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 14:47:22.680890 (+ 1178us) server_negotiation.cc:407] Connection header received
0620 14:47:22.681109 (+ 219us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:47:22.681115 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:47:22.681214 (+ 99us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:47:22.681327 (+ 113us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 14:47:22.681834 (+ 507us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:47:22.682993 (+ 1159us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:47:22.684543 (+ 1550us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:47:22.685219 (+ 676us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:47:22.685365 (+ 146us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:47:22.685776 (+ 411us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 14:47:22.685924 (+ 148us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0620 14:47:22.686019 (+ 95us) server_negotiation.cc:1035] Waiting for connection context
0620 14:47:22.686144 (+ 125us) server_negotiation.cc:299] Negotiation successful
0620 14:47:22.686227 (+ 83us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":49}
I20260620 14:47:22.686595 2082 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:47:22.679538 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.3:45431 (local address 127.0.97.3:46915)
0620 14:47:22.680775 (+ 1237us) negotiation.cc:107] Waiting for socket to connect
0620 14:47:22.680792 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0620 14:47:22.680965 (+ 173us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 14:47:22.681366 (+ 401us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 14:47:22.681370 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 14:47:22.681380 (+ 10us) client_negotiation.cc:189] Negotiated authn=TOKEN
0620 14:47:22.681690 (+ 310us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:47:22.681698 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:47:22.683153 (+ 1455us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:47:22.683156 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:47:22.684403 (+ 1247us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:47:22.684414 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:47:22.685411 (+ 997us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:47:22.685415 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:47:22.685529 (+ 114us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:47:22.685584 (+ 55us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0620 14:47:22.686028 (+ 444us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0620 14:47:22.686036 (+ 8us) client_negotiation.cc:769] Sending connection context
0620 14:47:22.686098 (+ 62us) client_negotiation.cc:240] Negotiation successful
0620 14:47:22.686212 (+ 114us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1065,"thread_start_us":898,"threads_started":1}
I20260620 14:47:22.760612 1511 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:56570:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 14:47:22.760885 1511 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:56570:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 14:47:22.763979 1511 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 8f6b9a65e0374b0b9f1f530836785c11: Sending DeleteTablet for 3 replicas of tablet 4488471c33004c6aa721be98c24846d9
I20260620 14:47:22.764916 1649 tablet_service.cc:1514] Processing DeleteTablet for tablet 4488471c33004c6aa721be98c24846d9 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 14:47:22 UTC) from {username='kudu', principal='kudu/127.0.97.62@KRBTEST.COM'} at 127.0.0.1:56206
I20260620 14:47:22.765045 1790 tablet_service.cc:1514] Processing DeleteTablet for tablet 4488471c33004c6aa721be98c24846d9 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 14:47:22 UTC) from {username='kudu', principal='kudu/127.0.97.62@KRBTEST.COM'} at 127.0.0.1:47260
I20260620 14:47:22.765646 2086 tablet_replica.cc:330] T 4488471c33004c6aa721be98c24846d9 P 796b41dc7b034f68a91a167e83106695: stopping tablet replica
I20260620 14:47:22.765802 2087 tablet_replica.cc:330] T 4488471c33004c6aa721be98c24846d9 P 58089d127abc4bc9b8e27348383b182d: stopping tablet replica
I20260620 14:47:22.765880 2086 raft_consensus.cc:2229] T 4488471c33004c6aa721be98c24846d9 P 796b41dc7b034f68a91a167e83106695 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 14:47:22.766007 2087 raft_consensus.cc:2229] T 4488471c33004c6aa721be98c24846d9 P 58089d127abc4bc9b8e27348383b182d [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 14:47:22.766273 2087 raft_consensus.cc:2258] T 4488471c33004c6aa721be98c24846d9 P 58089d127abc4bc9b8e27348383b182d [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 14:47:22.766335 2086 raft_consensus.cc:2258] T 4488471c33004c6aa721be98c24846d9 P 796b41dc7b034f68a91a167e83106695 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 14:47:22.767371 2087 ts_tablet_manager.cc:1905] T 4488471c33004c6aa721be98c24846d9 P 58089d127abc4bc9b8e27348383b182d: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 14:47:22.767498 1928 tablet_service.cc:1514] Processing DeleteTablet for tablet 4488471c33004c6aa721be98c24846d9 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 14:47:22 UTC) from {username='kudu', principal='kudu/127.0.97.62@KRBTEST.COM'} at 127.0.0.1:41162
I20260620 14:47:22.768196 2089 tablet_replica.cc:330] T 4488471c33004c6aa721be98c24846d9 P ed946240f9ce40fbb23487b38915e4e5: stopping tablet replica
I20260620 14:47:22.768445 2089 raft_consensus.cc:2229] T 4488471c33004c6aa721be98c24846d9 P ed946240f9ce40fbb23487b38915e4e5 [term 1 LEADER]: Raft consensus shutting down.
I20260620 14:47:22.769030 2089 raft_consensus.cc:2258] T 4488471c33004c6aa721be98c24846d9 P ed946240f9ce40fbb23487b38915e4e5 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 14:47:22.770172 2089 ts_tablet_manager.cc:1905] T 4488471c33004c6aa721be98c24846d9 P ed946240f9ce40fbb23487b38915e4e5: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 14:47:22.774395 2087 ts_tablet_manager.cc:1918] T 4488471c33004c6aa721be98c24846d9 P 58089d127abc4bc9b8e27348383b182d: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260620 14:47:22.774665 2087 log.cc:1196] T 4488471c33004c6aa721be98c24846d9 P 58089d127abc4bc9b8e27348383b182d: Deleting WAL directory at /tmp/dist-test-taskUci3DU/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781966829091300-388-0/minicluster-data/ts-1/wal/wals/4488471c33004c6aa721be98c24846d9
I20260620 14:47:22.775231 2087 ts_tablet_manager.cc:1939] T 4488471c33004c6aa721be98c24846d9 P 58089d127abc4bc9b8e27348383b182d: Deleting consensus metadata
I20260620 14:47:22.776556 1500 catalog_manager.cc:4841] TS 58089d127abc4bc9b8e27348383b182d (127.0.97.2:45941): tablet 4488471c33004c6aa721be98c24846d9 (table test-table [id=c1ed5eedb5294050aa692f14f371e214]) successfully deleted
I20260620 14:47:22.777782 2089 ts_tablet_manager.cc:1918] T 4488471c33004c6aa721be98c24846d9 P ed946240f9ce40fbb23487b38915e4e5: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260620 14:47:22.777956 2089 log.cc:1196] T 4488471c33004c6aa721be98c24846d9 P ed946240f9ce40fbb23487b38915e4e5: Deleting WAL directory at /tmp/dist-test-taskUci3DU/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781966829091300-388-0/minicluster-data/ts-2/wal/wals/4488471c33004c6aa721be98c24846d9
I20260620 14:47:22.778777 2089 ts_tablet_manager.cc:1939] T 4488471c33004c6aa721be98c24846d9 P ed946240f9ce40fbb23487b38915e4e5: Deleting consensus metadata
I20260620 14:47:22.779754 1497 catalog_manager.cc:4841] TS ed946240f9ce40fbb23487b38915e4e5 (127.0.97.3:45431): tablet 4488471c33004c6aa721be98c24846d9 (table test-table [id=c1ed5eedb5294050aa692f14f371e214]) successfully deleted
I20260620 14:47:22.780274 2086 ts_tablet_manager.cc:1905] T 4488471c33004c6aa721be98c24846d9 P 796b41dc7b034f68a91a167e83106695: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 14:47:22.788345 2086 ts_tablet_manager.cc:1918] T 4488471c33004c6aa721be98c24846d9 P 796b41dc7b034f68a91a167e83106695: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260620 14:47:22.788491 2086 log.cc:1196] T 4488471c33004c6aa721be98c24846d9 P 796b41dc7b034f68a91a167e83106695: Deleting WAL directory at /tmp/dist-test-taskUci3DU/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781966829091300-388-0/minicluster-data/ts-0/wal/wals/4488471c33004c6aa721be98c24846d9
I20260620 14:47:22.789064 2086 ts_tablet_manager.cc:1939] T 4488471c33004c6aa721be98c24846d9 P 796b41dc7b034f68a91a167e83106695: Deleting consensus metadata
I20260620 14:47:22.790637 1499 catalog_manager.cc:4841] TS 796b41dc7b034f68a91a167e83106695 (127.0.97.1:45961): tablet 4488471c33004c6aa721be98c24846d9 (table test-table [id=c1ed5eedb5294050aa692f14f371e214]) successfully deleted
Jun 20 14:47:22 dist-test-slave-q9j6 krb5kdc[1468](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781966841, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.0.97.1@KRBTEST.COM
I20260620 14:47:22.801564 2090 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:47:22.768647 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56212 (local address 127.0.97.1:45961)
0620 14:47:22.784618 (+ 15971us) server_negotiation.cc:206] Beginning negotiation
0620 14:47:22.784623 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 14:47:22.784639 (+ 16us) server_negotiation.cc:407] Connection header received
0620 14:47:22.784688 (+ 49us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:47:22.784691 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:47:22.784752 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:47:22.784892 (+ 140us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:47:22.786369 (+ 1477us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:47:22.787190 (+ 821us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:47:22.788597 (+ 1407us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:47:22.789068 (+ 471us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:47:22.789218 (+ 150us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:47:22.794292 (+ 5074us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 14:47:22.794307 (+ 15us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 14:47:22.794327 (+ 20us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 14:47:22.794368 (+ 41us) server_negotiation.cc:922] calling sasl_server_start()
0620 14:47:22.798257 (+ 3889us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:47:22.799181 (+ 924us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:47:22.799191 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:47:22.799195 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:47:22.799353 (+ 158us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:47:22.799829 (+ 476us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:47:22.799834 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:47:22.799837 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:47:22.800279 (+ 442us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 14:47:22.800563 (+ 284us) server_negotiation.cc:1035] Waiting for connection context
0620 14:47:22.800875 (+ 312us) server_negotiation.cc:299] Negotiation successful
0620 14:47:22.801095 (+ 220us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":15856,"thread_start_us":15772,"threads_started":1}
W20260620 14:47:22.802621 1715 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.SetFlag from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:56212
I20260620 14:47:22.832654 2094 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:47:22.807696 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56584 (local address 127.0.97.62:37781)
0620 14:47:22.814241 (+ 6545us) server_negotiation.cc:206] Beginning negotiation
0620 14:47:22.814248 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0620 14:47:22.817753 (+ 3505us) server_negotiation.cc:407] Connection header received
0620 14:47:22.817948 (+ 195us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:47:22.817953 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:47:22.818034 (+ 81us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:47:22.818154 (+ 120us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:47:22.820400 (+ 2246us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:47:22.821184 (+ 784us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:47:22.823006 (+ 1822us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:47:22.823394 (+ 388us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:47:22.823487 (+ 93us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:47:22.826414 (+ 2927us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 14:47:22.826419 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 14:47:22.826422 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 14:47:22.826466 (+ 44us) server_negotiation.cc:922] calling sasl_server_start()
0620 14:47:22.829908 (+ 3442us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:47:22.830606 (+ 698us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:47:22.830612 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:47:22.830616 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:47:22.830823 (+ 207us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:47:22.831173 (+ 350us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:47:22.831178 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:47:22.831181 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:47:22.831531 (+ 350us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 14:47:22.831813 (+ 282us) server_negotiation.cc:1035] Waiting for connection context
0620 14:47:22.832088 (+ 275us) server_negotiation.cc:299] Negotiation successful
0620 14:47:22.832318 (+ 230us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":6401,"thread_start_us":6247,"threads_started":1}
W20260620 14:47:22.833371 1511 server_base.cc:1116] Unauthorized access attempt to method kudu.master.MasterService.TSHeartbeat from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:56584
I20260620 14:47:22.835533 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskUci3DU/build/asan/bin/kudu pid 1584
I20260620 14:47:22.851438 2090 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:47:22.836990 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56220 (local address 127.0.97.1:45961)
0620 14:47:22.837206 (+ 216us) server_negotiation.cc:206] Beginning negotiation
0620 14:47:22.837212 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 14:47:22.838261 (+ 1049us) server_negotiation.cc:407] Connection header received
0620 14:47:22.838404 (+ 143us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:47:22.838410 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:47:22.838527 (+ 117us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:47:22.838627 (+ 100us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:47:22.839754 (+ 1127us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:47:22.840821 (+ 1067us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:47:22.842254 (+ 1433us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:47:22.842893 (+ 639us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:47:22.843063 (+ 170us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:47:22.845501 (+ 2438us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 14:47:22.845508 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 14:47:22.845513 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 14:47:22.845554 (+ 41us) server_negotiation.cc:922] calling sasl_server_start()
0620 14:47:22.848793 (+ 3239us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:47:22.849470 (+ 677us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:47:22.849478 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:47:22.849482 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:47:22.849654 (+ 172us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:47:22.850006 (+ 352us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:47:22.850012 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:47:22.850017 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:47:22.850347 (+ 330us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 14:47:22.850617 (+ 270us) server_negotiation.cc:1035] Waiting for connection context
0620 14:47:22.850816 (+ 199us) server_negotiation.cc:299] Negotiation successful
0620 14:47:22.851057 (+ 241us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":84}
W20260620 14:47:22.852401 1715 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:56220
W20260620 14:47:22.852959 388 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskUci3DU/build/asan/bin/kudu pid 1584: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260620 14:47:22.853160 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskUci3DU/build/asan/bin/kudu with pid 1584
I20260620 14:47:22.870821 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskUci3DU/build/asan/bin/kudu pid 1724
Jun 20 14:47:22 dist-test-slave-q9j6 krb5kdc[1468](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781966841, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.0.97.2@KRBTEST.COM
I20260620 14:47:22.886814 2096 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:47:22.871941 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47276 (local address 127.0.97.2:45941)
0620 14:47:22.872687 (+ 746us) server_negotiation.cc:206] Beginning negotiation
0620 14:47:22.872691 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 14:47:22.872707 (+ 16us) server_negotiation.cc:407] Connection header received
0620 14:47:22.872765 (+ 58us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:47:22.872769 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:47:22.872824 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:47:22.872958 (+ 134us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:47:22.874074 (+ 1116us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:47:22.874903 (+ 829us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:47:22.876269 (+ 1366us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:47:22.876697 (+ 428us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:47:22.876834 (+ 137us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:47:22.880989 (+ 4155us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 14:47:22.881003 (+ 14us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 14:47:22.881018 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 14:47:22.881056 (+ 38us) server_negotiation.cc:922] calling sasl_server_start()
0620 14:47:22.883945 (+ 2889us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:47:22.884628 (+ 683us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:47:22.884635 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:47:22.884638 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:47:22.884777 (+ 139us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:47:22.885160 (+ 383us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:47:22.885166 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:47:22.885169 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:47:22.885623 (+ 454us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 14:47:22.885917 (+ 294us) server_negotiation.cc:1035] Waiting for connection context
0620 14:47:22.886212 (+ 295us) server_negotiation.cc:299] Negotiation successful
0620 14:47:22.886457 (+ 245us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":640,"thread_start_us":539,"threads_started":1}
W20260620 14:47:22.887852 1855 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:47276
W20260620 14:47:22.888527 388 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskUci3DU/build/asan/bin/kudu pid 1724: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260620 14:47:22.888679 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskUci3DU/build/asan/bin/kudu with pid 1724
I20260620 14:47:22.917857 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskUci3DU/build/asan/bin/kudu pid 1864
Jun 20 14:47:22 dist-test-slave-q9j6 krb5kdc[1468](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781966841, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.0.97.3@KRBTEST.COM
I20260620 14:47:22.936724 2065 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:47:22.918680 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41180 (local address 127.0.97.3:45431)
0620 14:47:22.918857 (+ 177us) server_negotiation.cc:206] Beginning negotiation
0620 14:47:22.918863 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 14:47:22.918940 (+ 77us) server_negotiation.cc:407] Connection header received
0620 14:47:22.919142 (+ 202us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:47:22.919146 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:47:22.919222 (+ 76us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:47:22.919316 (+ 94us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:47:22.921078 (+ 1762us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:47:22.921798 (+ 720us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:47:22.923718 (+ 1920us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:47:22.924166 (+ 448us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:47:22.924278 (+ 112us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:47:22.929338 (+ 5060us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 14:47:22.929371 (+ 33us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 14:47:22.929393 (+ 22us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 14:47:22.929444 (+ 51us) server_negotiation.cc:922] calling sasl_server_start()
0620 14:47:22.933612 (+ 4168us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:47:22.934436 (+ 824us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:47:22.934446 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:47:22.934450 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:47:22.934626 (+ 176us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:47:22.935019 (+ 393us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:47:22.935024 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:47:22.935028 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:47:22.935551 (+ 523us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 14:47:22.935932 (+ 381us) server_negotiation.cc:1035] Waiting for connection context
0620 14:47:22.936107 (+ 175us) server_negotiation.cc:299] Negotiation successful
0620 14:47:22.936352 (+ 245us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":64}
W20260620 14:47:22.937454 1994 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:41180
W20260620 14:47:22.938221 388 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskUci3DU/build/asan/bin/kudu pid 1864: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260620 14:47:22.938405 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskUci3DU/build/asan/bin/kudu with pid 1864
I20260620 14:47:22.958277 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskUci3DU/build/asan/bin/kudu pid 1481
I20260620 14:47:22.959427 1553 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 14:47:23.073390 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskUci3DU/build/asan/bin/kudu with pid 1481
2026-06-20T14:47:23Z chronyd exiting
[ OK ] SecurityITest.SmokeTestAsAuthorizedUser (4996 ms)
[ RUN ] SecurityITest.TxnSmokeWithDifferentUserTypes
Loading random data
Initializing database '/tmp/dist-test-taskUci3DU/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781966829091300-388-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 20 14:47:23 dist-test-slave-q9j6 krb5kdc[2100](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 20 14:47:23 dist-test-slave-q9j6 krb5kdc[2100](info): set up 2 sockets
Jun 20 14:47:23 dist-test-slave-q9j6 krb5kdc[2100](info): commencing operation
krb5kdc: starting...
W20260620 14:47:24.151721 388 mini_kdc.cc:121] Time spent starting KDC: real 1.035s user 0.004s sys 0.012s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskUci3DU/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781966829091300-388-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskUci3DU/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781966829091300-388-0/krb5kdc/test-user.keytab.
Jun 20 14:47:24 dist-test-slave-q9j6 krb5kdc[2100](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781966844, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-20T14:47:24Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T14:47:24Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.97.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskUci3DU/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781966829091300-388-0/krb5kdc/kudu_127.0.97.62.keytab.
Entry for principal kudu/127.0.97.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskUci3DU/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781966829091300-388-0/krb5kdc/kudu_127.0.97.62.keytab.
I20260620 14:47:24.341055 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskUci3DU/build/asan/bin/kudu
/tmp/dist-test-taskUci3DU/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskUci3DU/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781966829091300-388-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskUci3DU/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781966829091300-388-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskUci3DU/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781966829091300-388-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskUci3DU/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781966829091300-388-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:37975
--webserver_interface=127.0.97.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskUci3DU/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781966829091300-388-0/krb5kdc/kudu_127.0.97.62.keytab
--principal=kudu/127.0.97.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:45189
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.97.62:37975
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--txn_manager_enabled=true with env {KRB5CCNAME=/tmp/dist-test-taskUci3DU/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781966829091300-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskUci3DU/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781966829091300-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskUci3DU/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781966829091300-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 14:47:24.665433 2113 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 14:47:24.665740 2113 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 14:47:24.665902 2113 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 14:47:24.673295 2113 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 14:47:24.673416 2113 flags.cc:425] Enabled experimental flag: --txn_manager_enabled=true
W20260620 14:47:24.673512 2113 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 14:47:24.673628 2113 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 14:47:24.673710 2113 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 14:47:24.673803 2113 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 14:47:24.684005 2113 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:45189
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskUci3DU/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781966829091300-388-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskUci3DU/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781966829091300-388-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.97.62:37975
--txn_manager_enabled=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskUci3DU/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781966829091300-388-0/krb5kdc/kudu_127.0.97.62.keytab
--principal=kudu/127.0.97.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:37975
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskUci3DU/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781966829091300-388-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.97.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskUci3DU/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781966829091300-388-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 13:41:20 UTC on ccycloud.quasar-vdagoi.root.comops.site
build id 4457462
ASAN enabled
I20260620 14:47:24.685344 2113 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 14:47:24.686668 2113 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 14:47:24.698267 2121 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 14:47:24.698385 2119 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 14:47:24.698365 2118 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 14:47:24.699302 2113 server_base.cc:1034] running on GCE node
I20260620 14:47:24.700186 2113 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 14:47:24.701538 2113 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 14:47:24.702828 2113 hybrid_clock.cc:648] HybridClock initialized: now 1781966844702752 us; error 39 us; skew 500 ppm
I20260620 14:47:24.703195 2113 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 14:47:24 dist-test-slave-q9j6 krb5kdc[2100](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781966844, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 14:47:24.706377 2113 init.cc:377] Logged in from keytab as kudu/127.0.97.62@KRBTEST.COM (short username kudu)
I20260620 14:47:24.707973 2113 webserver.cc:460] Webserver started at http://127.0.97.62:43293/ using document root <none> and password file <none>
I20260620 14:47:24.708734 2113 fs_manager.cc:359] Metadata directory not provided
I20260620 14:47:24.708873 2113 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 14:47:24.709295 2113 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 14:47:24.712756 2113 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskUci3DU/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781966829091300-388-0/minicluster-data/master-0/data/instance:
uuid: "dbf70d79ae0b41f28a6637ed9ab0ecc4"
format_stamp: "Formatted at 2026-06-20 14:47:24 on dist-test-slave-q9j6"
server_key: "5941de35cd4f7b46be005ce425f17564"
server_key_iv: "835d0a0599511e9d6c17e8be785d65a8"
server_key_version: "encryptionkey@0"
I20260620 14:47:24.713594 2113 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskUci3DU/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781966829091300-388-0/minicluster-data/master-0/wal/instance:
uuid: "dbf70d79ae0b41f28a6637ed9ab0ecc4"
format_stamp: "Formatted at 2026-06-20 14:47:24 on dist-test-slave-q9j6"
server_key: "5941de35cd4f7b46be005ce425f17564"
server_key_iv: "835d0a0599511e9d6c17e8be785d65a8"
server_key_version: "encryptionkey@0"
I20260620 14:47:24.719007 2113 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.000s
I20260620 14:47:24.722609 2128 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 14:47:24.724017 2113 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260620 14:47:24.724222 2113 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskUci3DU/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781966829091300-388-0/minicluster-data/master-0/data,/tmp/dist-test-taskUci3DU/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781966829091300-388-0/minicluster-data/master-0/wal
uuid: "dbf70d79ae0b41f28a6637ed9ab0ecc4"
format_stamp: "Formatted at 2026-06-20 14:47:24 on dist-test-slave-q9j6"
server_key: "5941de35cd4f7b46be005ce425f17564"
server_key_iv: "835d0a0599511e9d6c17e8be785d65a8"
server_key_version: "encryptionkey@0"
I20260620 14:47:24.724452 2113 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskUci3DU/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781966829091300-388-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskUci3DU/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781966829091300-388-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskUci3DU/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781966829091300-388-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 14:47:24.743023 2113 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 14:47:24.747116 2113 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 14:47:24.747406 2113 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 14:47:24.769207 2113 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.62:37975
I20260620 14:47:24.769243 2189 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.62:37975 every 8 connection(s)
I20260620 14:47:24.770603 2113 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskUci3DU/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781966829091300-388-0/minicluster-data/master-0/data/info.pb
I20260620 14:47:24.774771 2190 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 14:47:24.779259 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskUci3DU/build/asan/bin/kudu as pid 2113
I20260620 14:47:24.779497 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskUci3DU/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781966829091300-388-0/minicluster-data/master-0/wal/instance
I20260620 14:47:24.780098 388 external_mini_cluster.cc:1428] Setting key 736bf41fe765516c942a76ce0fdb5f4e
I20260620 14:47:24.786402 2190 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P dbf70d79ae0b41f28a6637ed9ab0ecc4: Bootstrap starting.
I20260620 14:47:24.790426 2190 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P dbf70d79ae0b41f28a6637ed9ab0ecc4: Neither blocks nor log segments found. Creating new log.
I20260620 14:47:24.791747 2190 log.cc:824] T 00000000000000000000000000000000 P dbf70d79ae0b41f28a6637ed9ab0ecc4: Log is configured to *not* fsync() on all Append() calls
I20260620 14:47:24.795210 2190 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P dbf70d79ae0b41f28a6637ed9ab0ecc4: No bootstrap required, opened a new log
Jun 20 14:47:24 dist-test-slave-q9j6 krb5kdc[2100](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781966844, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260620 14:47:24.806205 2190 raft_consensus.cc:348] T 00000000000000000000000000000000 P dbf70d79ae0b41f28a6637ed9ab0ecc4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "dbf70d79ae0b41f28a6637ed9ab0ecc4" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 37975 } }
I20260620 14:47:24.806661 2190 raft_consensus.cc:374] T 00000000000000000000000000000000 P dbf70d79ae0b41f28a6637ed9ab0ecc4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 14:47:24.806772 2190 raft_consensus.cc:729] T 00000000000000000000000000000000 P dbf70d79ae0b41f28a6637ed9ab0ecc4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: dbf70d79ae0b41f28a6637ed9ab0ecc4, State: Initialized, Role: FOLLOWER
I20260620 14:47:24.807281 2193 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:47:24.782435 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40398 (local address 127.0.97.62:37975)
0620 14:47:24.783490 (+ 1055us) server_negotiation.cc:206] Beginning negotiation
0620 14:47:24.783513 (+ 23us) server_negotiation.cc:399] Waiting for connection header
0620 14:47:24.784064 (+ 551us) server_negotiation.cc:407] Connection header received
0620 14:47:24.786538 (+ 2474us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:47:24.786569 (+ 31us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:47:24.787119 (+ 550us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:47:24.787626 (+ 507us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:47:24.788990 (+ 1364us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:47:24.791033 (+ 2043us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:47:24.792637 (+ 1604us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:47:24.793534 (+ 897us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:47:24.793713 (+ 179us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:47:24.798094 (+ 4381us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 14:47:24.798104 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 14:47:24.798130 (+ 26us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 14:47:24.798187 (+ 57us) server_negotiation.cc:922] calling sasl_server_start()
0620 14:47:24.802332 (+ 4145us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:47:24.803290 (+ 958us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:47:24.803300 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:47:24.803314 (+ 14us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:47:24.803536 (+ 222us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:47:24.803958 (+ 422us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:47:24.803964 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:47:24.803968 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:47:24.804716 (+ 748us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 14:47:24.805030 (+ 314us) server_negotiation.cc:1035] Waiting for connection context
0620 14:47:24.805401 (+ 371us) server_negotiation.cc:299] Negotiation successful
0620 14:47:24.805851 (+ 450us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":770,"thread_start_us":472,"threads_started":1}
I20260620 14:47:24.807469 2190 consensus_queue.cc:260] T 00000000000000000000000000000000 P dbf70d79ae0b41f28a6637ed9ab0ecc4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "dbf70d79ae0b41f28a6637ed9ab0ecc4" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 37975 } }
I20260620 14:47:24.807916 2190 raft_consensus.cc:388] T 00000000000000000000000000000000 P dbf70d79ae0b41f28a6637ed9ab0ecc4 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 14:47:24.808116 2190 raft_consensus.cc:482] T 00000000000000000000000000000000 P dbf70d79ae0b41f28a6637ed9ab0ecc4 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 14:47:24.808322 2190 raft_consensus.cc:3037] T 00000000000000000000000000000000 P dbf70d79ae0b41f28a6637ed9ab0ecc4 [term 0 FOLLOWER]: Advancing to term 1
I20260620 14:47:24.810688 2190 raft_consensus.cc:504] T 00000000000000000000000000000000 P dbf70d79ae0b41f28a6637ed9ab0ecc4 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "dbf70d79ae0b41f28a6637ed9ab0ecc4" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 37975 } }
I20260620 14:47:24.811180 2190 leader_election.cc:302] T 00000000000000000000000000000000 P dbf70d79ae0b41f28a6637ed9ab0ecc4 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: dbf70d79ae0b41f28a6637ed9ab0ecc4; no voters:
I20260620 14:47:24.811805 2190 leader_election.cc:288] T 00000000000000000000000000000000 P dbf70d79ae0b41f28a6637ed9ab0ecc4 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260620 14:47:24.811929 2195 raft_consensus.cc:2781] T 00000000000000000000000000000000 P dbf70d79ae0b41f28a6637ed9ab0ecc4 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 14:47:24.812777 2195 raft_consensus.cc:686] T 00000000000000000000000000000000 P dbf70d79ae0b41f28a6637ed9ab0ecc4 [term 1 LEADER]: Becoming Leader. State: Replica: dbf70d79ae0b41f28a6637ed9ab0ecc4, State: Running, Role: LEADER
I20260620 14:47:24.813378 2195 consensus_queue.cc:237] T 00000000000000000000000000000000 P dbf70d79ae0b41f28a6637ed9ab0ecc4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "dbf70d79ae0b41f28a6637ed9ab0ecc4" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 37975 } }
I20260620 14:47:24.814016 2190 sys_catalog.cc:564] T 00000000000000000000000000000000 P dbf70d79ae0b41f28a6637ed9ab0ecc4 [sys.catalog]: configured and running, proceeding with master startup.
I20260620 14:47:24.816321 2197 sys_catalog.cc:455] T 00000000000000000000000000000000 P dbf70d79ae0b41f28a6637ed9ab0ecc4 [sys.catalog]: SysCatalogTable state changed. Reason: New leader dbf70d79ae0b41f28a6637ed9ab0ecc4. Latest consensus state: current_term: 1 leader_uuid: "dbf70d79ae0b41f28a6637ed9ab0ecc4" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "dbf70d79ae0b41f28a6637ed9ab0ecc4" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 37975 } } }
I20260620 14:47:24.816769 2197 sys_catalog.cc:458] T 00000000000000000000000000000000 P dbf70d79ae0b41f28a6637ed9ab0ecc4 [sys.catalog]: This master's current role is: LEADER
I20260620 14:47:24.820876 2203 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 14:47:24.820804 2196 sys_catalog.cc:455] T 00000000000000000000000000000000 P dbf70d79ae0b41f28a6637ed9ab0ecc4 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "dbf70d79ae0b41f28a6637ed9ab0ecc4" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "dbf70d79ae0b41f28a6637ed9ab0ecc4" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 37975 } } }
I20260620 14:47:24.821110 2196 sys_catalog.cc:458] T 00000000000000000000000000000000 P dbf70d79ae0b41f28a6637ed9ab0ecc4 [sys.catalog]: This master's current role is: LEADER
I20260620 14:47:24.826116 2203 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 14:47:24.835041 2203 catalog_manager.cc:1347] Generated new cluster ID: b8474891e98440d989c5f6d3a7e334e9
I20260620 14:47:24.835160 2203 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 14:47:24.861797 2203 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 14:47:24.863181 2203 catalog_manager.cc:1504] Loading token signing keys...
I20260620 14:47:24.871999 2203 catalog_manager.cc:5868] T 00000000000000000000000000000000 P dbf70d79ae0b41f28a6637ed9ab0ecc4: Generated new TSK 0
I20260620 14:47:24.873014 2203 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.0.97.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskUci3DU/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781966829091300-388-0/krb5kdc/kudu_127.0.97.1.keytab.
Entry for principal kudu/127.0.97.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskUci3DU/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781966829091300-388-0/krb5kdc/kudu_127.0.97.1.keytab.
I20260620 14:47:24.934587 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskUci3DU/build/asan/bin/kudu
/tmp/dist-test-taskUci3DU/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskUci3DU/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781966829091300-388-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskUci3DU/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781966829091300-388-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskUci3DU/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781966829091300-388-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskUci3DU/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781966829091300-388-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.1:0
--local_ip_for_outbound_sockets=127.0.97.1
--webserver_interface=127.0.97.1
--webserver_port=0
--tserver_master_addrs=127.0.97.62:37975
--keytab_file=/tmp/dist-test-taskUci3DU/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781966829091300-388-0/krb5kdc/kudu_127.0.97.1.keytab
--principal=kudu/127.0.97.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:45189
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true
--txn_keepalive_interval_ms=500 with env {KRB5CCNAME=/tmp/dist-test-taskUci3DU/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781966829091300-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskUci3DU/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781966829091300-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskUci3DU/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781966829091300-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 14:47:25.270826 2216 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 14:47:25.271145 2216 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 14:47:25.271315 2216 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 14:47:25.278879 2216 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 14:47:25.279007 2216 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 14:47:25.279172 2216 flags.cc:425] Enabled experimental flag: --txn_keepalive_interval_ms=500
W20260620 14:47:25.279255 2216 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260620 14:47:25.279385 2216 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.1
I20260620 14:47:25.289664 2216 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:45189
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskUci3DU/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781966829091300-388-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskUci3DU/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781966829091300-388-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskUci3DU/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781966829091300-388-0/krb5kdc/kudu_127.0.97.1.keytab
--principal=kudu/127.0.97.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskUci3DU/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781966829091300-388-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.97.1
--webserver_port=0
--txn_keepalive_interval_ms=500
--enable_txn_system_client_init=true
--tserver_master_addrs=127.0.97.62:37975
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.1
--log_dir=/tmp/dist-test-taskUci3DU/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781966829091300-388-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 13:41:20 UTC on ccycloud.quasar-vdagoi.root.comops.site
build id 4457462
ASAN enabled
I20260620 14:47:25.291049 2216 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 14:47:25.292532 2216 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 14:47:25.305892 2222 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 14:47:25.305869 2224 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 14:47:25.306275 2221 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 14:47:25.306710 2216 server_base.cc:1034] running on GCE node
I20260620 14:47:25.307334 2216 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 14:47:25.308336 2216 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 14:47:25.309542 2216 hybrid_clock.cc:648] HybridClock initialized: now 1781966845309485 us; error 52 us; skew 500 ppm
I20260620 14:47:25.309866 2216 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 14:47:25 dist-test-slave-q9j6 krb5kdc[2100](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781966845, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 14:47:25.317065 2216 init.cc:377] Logged in from keytab as kudu/127.0.97.1@KRBTEST.COM (short username kudu)
I20260620 14:47:25.318635 2216 webserver.cc:460] Webserver started at http://127.0.97.1:36665/ using document root <none> and password file <none>
I20260620 14:47:25.319319 2216 fs_manager.cc:359] Metadata directory not provided
I20260620 14:47:25.319420 2216 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 14:47:25.319751 2216 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 14:47:25.322849 2216 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskUci3DU/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781966829091300-388-0/minicluster-data/ts-0/data/instance:
uuid: "5aa58728f3b74083b017cdcd573bc626"
format_stamp: "Formatted at 2026-06-20 14:47:25 on dist-test-slave-q9j6"
server_key: "6a72a875a731230b0b1727bc1c4632af"
server_key_iv: "7cc0a9ba5e13f72bc0ecf7adb7af67aa"
server_key_version: "encryptionkey@0"
I20260620 14:47:25.323724 2216 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskUci3DU/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781966829091300-388-0/minicluster-data/ts-0/wal/instance:
uuid: "5aa58728f3b74083b017cdcd573bc626"
format_stamp: "Formatted at 2026-06-20 14:47:25 on dist-test-slave-q9j6"
server_key: "6a72a875a731230b0b1727bc1c4632af"
server_key_iv: "7cc0a9ba5e13f72bc0ecf7adb7af67aa"
server_key_version: "encryptionkey@0"
I20260620 14:47:25.328851 2216 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.003s sys 0.001s
I20260620 14:47:25.332261 2231 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 14:47:25.333575 2216 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260620 14:47:25.333721 2216 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskUci3DU/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781966829091300-388-0/minicluster-data/ts-0/data,/tmp/dist-test-taskUci3DU/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781966829091300-388-0/minicluster-data/ts-0/wal
uuid: "5aa58728f3b74083b017cdcd573bc626"
format_stamp: "Formatted at 2026-06-20 14:47:25 on dist-test-slave-q9j6"
server_key: "6a72a875a731230b0b1727bc1c4632af"
server_key_iv: "7cc0a9ba5e13f72bc0ecf7adb7af67aa"
server_key_version: "encryptionkey@0"
I20260620 14:47:25.333909 2216 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskUci3DU/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781966829091300-388-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskUci3DU/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781966829091300-388-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskUci3DU/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781966829091300-388-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 14:47:25.354664 2216 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 14:47:25.360314 2216 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 14:47:25.360729 2216 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 14:47:25.364298 2216 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 14:47:25.364419 2216 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 14:47:25.364538 2216 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 14:47:25.364630 2216 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Jun 20 14:47:25 dist-test-slave-q9j6 krb5kdc[2100](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781966845, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.1@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260620 14:47:25.393328 2241 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:47:25.365677 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:37975 (local address 127.0.97.1:41917)
0620 14:47:25.366811 (+ 1134us) negotiation.cc:107] Waiting for socket to connect
0620 14:47:25.366882 (+ 71us) client_negotiation.cc:174] Beginning negotiation
0620 14:47:25.369296 (+ 2414us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 14:47:25.370361 (+ 1065us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 14:47:25.370376 (+ 15us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 14:47:25.371572 (+ 1196us) client_negotiation.cc:189] Negotiated authn=SASL
0620 14:47:25.372965 (+ 1393us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:47:25.373003 (+ 38us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:47:25.374883 (+ 1880us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:47:25.374894 (+ 11us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:47:25.377188 (+ 2294us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:47:25.377206 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:47:25.378439 (+ 1233us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:47:25.378444 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:47:25.378671 (+ 227us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:47:25.379545 (+ 874us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 14:47:25.379592 (+ 47us) client_negotiation.cc:656] calling sasl_client_start()
0620 14:47:25.383064 (+ 3472us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 14:47:25.387636 (+ 4572us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 14:47:25.387652 (+ 16us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 14:47:25.387684 (+ 32us) client_negotiation.cc:760] calling sasl_client_step()
0620 14:47:25.388380 (+ 696us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 14:47:25.388870 (+ 490us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 14:47:25.388879 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 14:47:25.388885 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0620 14:47:25.389225 (+ 340us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 14:47:25.389834 (+ 609us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 14:47:25.389843 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 14:47:25.390572 (+ 729us) client_negotiation.cc:769] Sending connection context
0620 14:47:25.390990 (+ 418us) client_negotiation.cc:240] Negotiation successful
0620 14:47:25.391666 (+ 676us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":725,"thread_start_us":435,"threads_started":1}
I20260620 14:47:25.394452 2240 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:47:25.364907 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.1:41917 (local address 127.0.97.62:37975)
0620 14:47:25.367530 (+ 2623us) server_negotiation.cc:206] Beginning negotiation
0620 14:47:25.367537 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0620 14:47:25.367562 (+ 25us) server_negotiation.cc:407] Connection header received
0620 14:47:25.369852 (+ 2290us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:47:25.369858 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:47:25.369949 (+ 91us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:47:25.370082 (+ 133us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:47:25.373243 (+ 3161us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:47:25.374495 (+ 1252us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:47:25.377374 (+ 2879us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:47:25.377964 (+ 590us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:47:25.378138 (+ 174us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:47:25.383366 (+ 5228us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 14:47:25.383375 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 14:47:25.383382 (+ 7us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 14:47:25.383436 (+ 54us) server_negotiation.cc:922] calling sasl_server_start()
0620 14:47:25.387417 (+ 3981us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:47:25.388518 (+ 1101us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:47:25.388524 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:47:25.388528 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:47:25.388705 (+ 177us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:47:25.389373 (+ 668us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:47:25.389378 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:47:25.389382 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:47:25.389687 (+ 305us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 14:47:25.389848 (+ 161us) server_negotiation.cc:1035] Waiting for connection context
0620 14:47:25.393970 (+ 4122us) server_negotiation.cc:299] Negotiation successful
0620 14:47:25.394153 (+ 183us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":2487,"thread_start_us":2374,"threads_started":1}
I20260620 14:47:25.415947 2308 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:47:25.400054 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:37975 (local address 127.0.97.1:53027)
0620 14:47:25.400829 (+ 775us) negotiation.cc:107] Waiting for socket to connect
0620 14:47:25.400845 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0620 14:47:25.401011 (+ 166us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 14:47:25.401355 (+ 344us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 14:47:25.401363 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 14:47:25.402118 (+ 755us) client_negotiation.cc:189] Negotiated authn=SASL
0620 14:47:25.402615 (+ 497us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:47:25.402633 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:47:25.403975 (+ 1342us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:47:25.403979 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:47:25.405762 (+ 1783us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:47:25.405783 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:47:25.406756 (+ 973us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:47:25.406761 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:47:25.406926 (+ 165us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:47:25.407560 (+ 634us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 14:47:25.407578 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0620 14:47:25.408530 (+ 952us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 14:47:25.412295 (+ 3765us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 14:47:25.412301 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 14:47:25.412305 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0620 14:47:25.412948 (+ 643us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 14:47:25.413414 (+ 466us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 14:47:25.413424 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 14:47:25.413429 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0620 14:47:25.413564 (+ 135us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 14:47:25.414267 (+ 703us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 14:47:25.414272 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 14:47:25.414680 (+ 408us) client_negotiation.cc:769] Sending connection context
0620 14:47:25.414870 (+ 190us) client_negotiation.cc:240] Negotiation successful
0620 14:47:25.415228 (+ 358us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":650,"thread_start_us":515,"threads_started":1}
I20260620 14:47:25.416275 2240 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:47:25.400019 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.1:53027 (local address 127.0.97.62:37975)
0620 14:47:25.400195 (+ 176us) server_negotiation.cc:206] Beginning negotiation
0620 14:47:25.400203 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0620 14:47:25.400937 (+ 734us) server_negotiation.cc:407] Connection header received
0620 14:47:25.401154 (+ 217us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:47:25.401159 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:47:25.401234 (+ 75us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:47:25.401332 (+ 98us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:47:25.402841 (+ 1509us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:47:25.403799 (+ 958us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:47:25.405975 (+ 2176us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:47:25.406555 (+ 580us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:47:25.406699 (+ 144us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:47:25.408735 (+ 2036us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 14:47:25.408742 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 14:47:25.408747 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 14:47:25.408786 (+ 39us) server_negotiation.cc:922] calling sasl_server_start()
0620 14:47:25.412179 (+ 3393us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:47:25.413075 (+ 896us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:47:25.413081 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:47:25.413084 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:47:25.413261 (+ 177us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:47:25.413691 (+ 430us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:47:25.413698 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:47:25.413704 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:47:25.414131 (+ 427us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 14:47:25.414313 (+ 182us) server_negotiation.cc:1035] Waiting for connection context
0620 14:47:25.415512 (+ 1199us) server_negotiation.cc:299] Negotiation successful
0620 14:47:25.415849 (+ 337us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":38}
I20260620 14:47:25.419155 2216 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.1:35927
I20260620 14:47:25.419188 2351 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.1:35927 every 8 connection(s)
I20260620 14:47:25.421114 2216 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskUci3DU/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781966829091300-388-0/minicluster-data/ts-0/data/info.pb
I20260620 14:47:25.423256 2352 heartbeater.cc:347] Connected to a master server at 127.0.97.62:37975
I20260620 14:47:25.423446 2352 heartbeater.cc:464] Registering TS with master...
I20260620 14:47:25.424134 2352 heartbeater.cc:511] Master 127.0.97.62:37975 requested a full tablet report, sending...
I20260620 14:47:25.425940 2145 ts_manager.cc:194] Registered new tserver with Master: 5aa58728f3b74083b017cdcd573bc626 (127.0.97.1:35927)
I20260620 14:47:25.427048 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskUci3DU/build/asan/bin/kudu as pid 2216
I20260620 14:47:25.427270 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskUci3DU/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781966829091300-388-0/minicluster-data/ts-0/wal/instance
I20260620 14:47:25.427938 2145 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.97.1@KRBTEST.COM'} at 127.0.97.1:41917
I20260620 14:47:25.428053 388 external_mini_cluster.cc:1428] Setting key 4058825f8d1b0921213d0d96366c1885
WARNING: no policy specified for kudu/127.0.97.2@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskUci3DU/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781966829091300-388-0/krb5kdc/kudu_127.0.97.2.keytab.
Entry for principal kudu/127.0.97.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskUci3DU/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781966829091300-388-0/krb5kdc/kudu_127.0.97.2.keytab.
I20260620 14:47:25.465041 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskUci3DU/build/asan/bin/kudu
/tmp/dist-test-taskUci3DU/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskUci3DU/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781966829091300-388-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskUci3DU/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781966829091300-388-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskUci3DU/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781966829091300-388-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskUci3DU/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781966829091300-388-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.2:0
--local_ip_for_outbound_sockets=127.0.97.2
--webserver_interface=127.0.97.2
--webserver_port=0
--tserver_master_addrs=127.0.97.62:37975
--keytab_file=/tmp/dist-test-taskUci3DU/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781966829091300-388-0/krb5kdc/kudu_127.0.97.2.keytab
--principal=kudu/127.0.97.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:45189
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true
--txn_keepalive_interval_ms=500 with env {KRB5CCNAME=/tmp/dist-test-taskUci3DU/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781966829091300-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskUci3DU/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781966829091300-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskUci3DU/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781966829091300-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 14:47:25.804093 2356 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 14:47:25.804392 2356 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 14:47:25.804494 2356 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 14:47:25.812477 2356 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 14:47:25.812605 2356 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 14:47:25.812695 2356 flags.cc:425] Enabled experimental flag: --txn_keepalive_interval_ms=500
W20260620 14:47:25.812752 2356 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260620 14:47:25.812814 2356 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.2
I20260620 14:47:25.823974 2356 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:45189
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskUci3DU/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781966829091300-388-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskUci3DU/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781966829091300-388-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskUci3DU/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781966829091300-388-0/krb5kdc/kudu_127.0.97.2.keytab
--principal=kudu/127.0.97.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskUci3DU/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781966829091300-388-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.97.2
--webserver_port=0
--txn_keepalive_interval_ms=500
--enable_txn_system_client_init=true
--tserver_master_addrs=127.0.97.62:37975
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.2
--log_dir=/tmp/dist-test-taskUci3DU/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781966829091300-388-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 13:41:20 UTC on ccycloud.quasar-vdagoi.root.comops.site
build id 4457462
ASAN enabled
I20260620 14:47:25.825320 2356 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 14:47:25.826570 2356 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 14:47:25.840176 2361 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 14:47:25.840229 2362 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 14:47:25.840147 2364 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 14:47:25.840883 2356 server_base.cc:1034] running on GCE node
I20260620 14:47:25.841538 2356 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 14:47:25.842487 2356 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 14:47:25.843689 2356 hybrid_clock.cc:648] HybridClock initialized: now 1781966845843639 us; error 39 us; skew 500 ppm
I20260620 14:47:25.844020 2356 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 14:47:25 dist-test-slave-q9j6 krb5kdc[2100](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781966845, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 14:47:25.847193 2356 init.cc:377] Logged in from keytab as kudu/127.0.97.2@KRBTEST.COM (short username kudu)
I20260620 14:47:25.848692 2356 webserver.cc:460] Webserver started at http://127.0.97.2:36117/ using document root <none> and password file <none>
I20260620 14:47:25.849426 2356 fs_manager.cc:359] Metadata directory not provided
I20260620 14:47:25.849567 2356 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 14:47:25.849911 2356 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 14:47:25.852985 2356 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskUci3DU/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781966829091300-388-0/minicluster-data/ts-1/data/instance:
uuid: "e5717b124bc7489e8817304e3108a0eb"
format_stamp: "Formatted at 2026-06-20 14:47:25 on dist-test-slave-q9j6"
server_key: "c5df47e6df4387116a3f863c867fde59"
server_key_iv: "3f50177d09ed85975f90e02819d54d7d"
server_key_version: "encryptionkey@0"
I20260620 14:47:25.853830 2356 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskUci3DU/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781966829091300-388-0/minicluster-data/ts-1/wal/instance:
uuid: "e5717b124bc7489e8817304e3108a0eb"
format_stamp: "Formatted at 2026-06-20 14:47:25 on dist-test-slave-q9j6"
server_key: "c5df47e6df4387116a3f863c867fde59"
server_key_iv: "3f50177d09ed85975f90e02819d54d7d"
server_key_version: "encryptionkey@0"
I20260620 14:47:25.859383 2356 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.001s
I20260620 14:47:25.863022 2371 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 14:47:25.864461 2356 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260620 14:47:25.864636 2356 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskUci3DU/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781966829091300-388-0/minicluster-data/ts-1/data,/tmp/dist-test-taskUci3DU/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781966829091300-388-0/minicluster-data/ts-1/wal
uuid: "e5717b124bc7489e8817304e3108a0eb"
format_stamp: "Formatted at 2026-06-20 14:47:25 on dist-test-slave-q9j6"
server_key: "c5df47e6df4387116a3f863c867fde59"
server_key_iv: "3f50177d09ed85975f90e02819d54d7d"
server_key_version: "encryptionkey@0"
I20260620 14:47:25.864840 2356 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskUci3DU/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781966829091300-388-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskUci3DU/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781966829091300-388-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskUci3DU/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781966829091300-388-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 14:47:25.877656 2356 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 14:47:25.881986 2356 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 14:47:25.882280 2356 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 14:47:25.885246 2356 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 14:47:25.885371 2356 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 14:47:25.885500 2356 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 14:47:25.885607 2356 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Jun 20 14:47:25 dist-test-slave-q9j6 krb5kdc[2100](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781966845, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.2@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260620 14:47:25.916589 2240 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:47:25.885773 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.2:46539 (local address 127.0.97.62:37975)
0620 14:47:25.885996 (+ 223us) server_negotiation.cc:206] Beginning negotiation
0620 14:47:25.886005 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0620 14:47:25.887782 (+ 1777us) server_negotiation.cc:407] Connection header received
0620 14:47:25.890234 (+ 2452us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:47:25.890240 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:47:25.890344 (+ 104us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:47:25.890468 (+ 124us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:47:25.893635 (+ 3167us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:47:25.894826 (+ 1191us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:47:25.897691 (+ 2865us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:47:25.898418 (+ 727us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:47:25.898583 (+ 165us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:47:25.908468 (+ 9885us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 14:47:25.908477 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 14:47:25.908483 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 14:47:25.908537 (+ 54us) server_negotiation.cc:922] calling sasl_server_start()
0620 14:47:25.911954 (+ 3417us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:47:25.913292 (+ 1338us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:47:25.913296 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:47:25.913299 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:47:25.913420 (+ 121us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:47:25.914123 (+ 703us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:47:25.914131 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:47:25.914136 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:47:25.914570 (+ 434us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 14:47:25.914746 (+ 176us) server_negotiation.cc:1035] Waiting for connection context
0620 14:47:25.915996 (+ 1250us) server_negotiation.cc:299] Negotiation successful
0620 14:47:25.916207 (+ 211us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":62}
I20260620 14:47:25.918810 2380 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:47:25.886519 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:37975 (local address 127.0.97.2:46539)
0620 14:47:25.887474 (+ 955us) negotiation.cc:107] Waiting for socket to connect
0620 14:47:25.887516 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0620 14:47:25.889907 (+ 2391us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 14:47:25.890782 (+ 875us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 14:47:25.890799 (+ 17us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 14:47:25.892346 (+ 1547us) client_negotiation.cc:189] Negotiated authn=SASL
0620 14:47:25.893430 (+ 1084us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:47:25.893453 (+ 23us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:47:25.895076 (+ 1623us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:47:25.895084 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:47:25.897493 (+ 2409us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:47:25.897512 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:47:25.898741 (+ 1229us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:47:25.898752 (+ 11us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:47:25.899162 (+ 410us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:47:25.903888 (+ 4726us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 14:47:25.903933 (+ 45us) client_negotiation.cc:656] calling sasl_client_start()
0620 14:47:25.908080 (+ 4147us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 14:47:25.912202 (+ 4122us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 14:47:25.912216 (+ 14us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 14:47:25.912249 (+ 33us) client_negotiation.cc:760] calling sasl_client_step()
0620 14:47:25.913138 (+ 889us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 14:47:25.913544 (+ 406us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 14:47:25.913552 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 14:47:25.913559 (+ 7us) client_negotiation.cc:760] calling sasl_client_step()
0620 14:47:25.913966 (+ 407us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 14:47:25.914738 (+ 772us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 14:47:25.914751 (+ 13us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 14:47:25.915414 (+ 663us) client_negotiation.cc:769] Sending connection context
0620 14:47:25.915856 (+ 442us) client_negotiation.cc:240] Negotiation successful
0620 14:47:25.916646 (+ 790us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":683,"thread_start_us":429,"threads_started":1}
I20260620 14:47:25.942941 2461 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:47:25.928399 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:37975 (local address 127.0.97.2:46725)
0620 14:47:25.929106 (+ 707us) negotiation.cc:107] Waiting for socket to connect
0620 14:47:25.929126 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0620 14:47:25.929293 (+ 167us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 14:47:25.929840 (+ 547us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 14:47:25.929845 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 14:47:25.930527 (+ 682us) client_negotiation.cc:189] Negotiated authn=SASL
0620 14:47:25.931025 (+ 498us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:47:25.931045 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:47:25.932135 (+ 1090us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:47:25.932140 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:47:25.933686 (+ 1546us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:47:25.933699 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:47:25.934456 (+ 757us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:47:25.934463 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:47:25.934621 (+ 158us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:47:25.935363 (+ 742us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 14:47:25.935381 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0620 14:47:25.936342 (+ 961us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 14:47:25.939619 (+ 3277us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 14:47:25.939626 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 14:47:25.939631 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0620 14:47:25.940213 (+ 582us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 14:47:25.940656 (+ 443us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 14:47:25.940665 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 14:47:25.940672 (+ 7us) client_negotiation.cc:760] calling sasl_client_step()
0620 14:47:25.940880 (+ 208us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 14:47:25.941689 (+ 809us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 14:47:25.941699 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 14:47:25.941934 (+ 235us) client_negotiation.cc:769] Sending connection context
0620 14:47:25.942178 (+ 244us) client_negotiation.cc:240] Negotiation successful
0620 14:47:25.942571 (+ 393us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":563,"thread_start_us":420,"threads_started":1}
I20260620 14:47:25.943763 2240 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:47:25.928247 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.2:46725 (local address 127.0.97.62:37975)
0620 14:47:25.928451 (+ 204us) server_negotiation.cc:206] Beginning negotiation
0620 14:47:25.928457 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 14:47:25.929219 (+ 762us) server_negotiation.cc:407] Connection header received
0620 14:47:25.929448 (+ 229us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:47:25.929453 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:47:25.929537 (+ 84us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:47:25.929640 (+ 103us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:47:25.931196 (+ 1556us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:47:25.932000 (+ 804us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:47:25.933849 (+ 1849us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:47:25.934260 (+ 411us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:47:25.934390 (+ 130us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:47:25.936551 (+ 2161us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 14:47:25.936556 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 14:47:25.936559 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 14:47:25.936592 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0620 14:47:25.939466 (+ 2874us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:47:25.940398 (+ 932us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:47:25.940402 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:47:25.940404 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:47:25.940518 (+ 114us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:47:25.941035 (+ 517us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:47:25.941043 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:47:25.941047 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:47:25.941409 (+ 362us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 14:47:25.941590 (+ 181us) server_negotiation.cc:1035] Waiting for connection context
0620 14:47:25.943265 (+ 1675us) server_negotiation.cc:299] Negotiation successful
0620 14:47:25.943481 (+ 216us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":52}
I20260620 14:47:25.949193 2356 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.2:37155
I20260620 14:47:25.949236 2490 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.2:37155 every 8 connection(s)
I20260620 14:47:25.950487 2356 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskUci3DU/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781966829091300-388-0/minicluster-data/ts-1/data/info.pb
I20260620 14:47:25.952512 2491 heartbeater.cc:347] Connected to a master server at 127.0.97.62:37975
I20260620 14:47:25.952733 2491 heartbeater.cc:464] Registering TS with master...
I20260620 14:47:25.953382 2491 heartbeater.cc:511] Master 127.0.97.62:37975 requested a full tablet report, sending...
I20260620 14:47:25.953878 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskUci3DU/build/asan/bin/kudu as pid 2356
I20260620 14:47:25.954051 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskUci3DU/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781966829091300-388-0/minicluster-data/ts-1/wal/instance
I20260620 14:47:25.954600 388 external_mini_cluster.cc:1428] Setting key eff56dccf569ad3b4015ac16ac55f473
I20260620 14:47:25.954789 2145 ts_manager.cc:194] Registered new tserver with Master: e5717b124bc7489e8817304e3108a0eb (127.0.97.2:37155)
I20260620 14:47:25.955754 2145 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.97.2@KRBTEST.COM'} at 127.0.97.2:46539
WARNING: no policy specified for kudu/127.0.97.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskUci3DU/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781966829091300-388-0/krb5kdc/kudu_127.0.97.3.keytab.
Entry for principal kudu/127.0.97.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskUci3DU/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781966829091300-388-0/krb5kdc/kudu_127.0.97.3.keytab.
I20260620 14:47:26.014894 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskUci3DU/build/asan/bin/kudu
/tmp/dist-test-taskUci3DU/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskUci3DU/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781966829091300-388-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskUci3DU/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781966829091300-388-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskUci3DU/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781966829091300-388-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskUci3DU/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781966829091300-388-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.3:0
--local_ip_for_outbound_sockets=127.0.97.3
--webserver_interface=127.0.97.3
--webserver_port=0
--tserver_master_addrs=127.0.97.62:37975
--keytab_file=/tmp/dist-test-taskUci3DU/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781966829091300-388-0/krb5kdc/kudu_127.0.97.3.keytab
--principal=kudu/127.0.97.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:45189
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true
--txn_keepalive_interval_ms=500 with env {KRB5CCNAME=/tmp/dist-test-taskUci3DU/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781966829091300-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskUci3DU/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781966829091300-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskUci3DU/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781966829091300-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 14:47:26.346184 2495 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 14:47:26.346535 2495 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 14:47:26.346710 2495 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 14:47:26.354180 2495 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 14:47:26.354350 2495 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 14:47:26.354503 2495 flags.cc:425] Enabled experimental flag: --txn_keepalive_interval_ms=500
W20260620 14:47:26.354588 2495 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260620 14:47:26.354732 2495 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.3
I20260620 14:47:26.365043 2495 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:45189
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskUci3DU/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781966829091300-388-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskUci3DU/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781966829091300-388-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskUci3DU/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781966829091300-388-0/krb5kdc/kudu_127.0.97.3.keytab
--principal=kudu/127.0.97.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskUci3DU/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781966829091300-388-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.97.3
--webserver_port=0
--txn_keepalive_interval_ms=500
--enable_txn_system_client_init=true
--tserver_master_addrs=127.0.97.62:37975
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.3
--log_dir=/tmp/dist-test-taskUci3DU/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781966829091300-388-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 13:41:20 UTC on ccycloud.quasar-vdagoi.root.comops.site
build id 4457462
ASAN enabled
I20260620 14:47:26.366504 2495 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 14:47:26.367987 2495 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 14:47:26.382031 2501 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 14:47:26.382120 2500 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 14:47:26.382077 2503 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 14:47:26.383195 2495 server_base.cc:1034] running on GCE node
I20260620 14:47:26.383966 2495 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 14:47:26.385035 2495 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 14:47:26.386255 2495 hybrid_clock.cc:648] HybridClock initialized: now 1781966846386200 us; error 43 us; skew 500 ppm
I20260620 14:47:26.386605 2495 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 14:47:26 dist-test-slave-q9j6 krb5kdc[2100](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781966846, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 14:47:26.389927 2495 init.cc:377] Logged in from keytab as kudu/127.0.97.3@KRBTEST.COM (short username kudu)
I20260620 14:47:26.391506 2495 webserver.cc:460] Webserver started at http://127.0.97.3:39093/ using document root <none> and password file <none>
I20260620 14:47:26.392293 2495 fs_manager.cc:359] Metadata directory not provided
I20260620 14:47:26.392429 2495 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 14:47:26.392824 2495 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 14:47:26.395898 2495 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskUci3DU/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781966829091300-388-0/minicluster-data/ts-2/data/instance:
uuid: "7f1b44e3bb694105afe8822fc7cb348b"
format_stamp: "Formatted at 2026-06-20 14:47:26 on dist-test-slave-q9j6"
server_key: "4991f8785e8e1f2576f5283d21d7abd0"
server_key_iv: "d0062fae206815f0b3c69ae66e33dbf1"
server_key_version: "encryptionkey@0"
I20260620 14:47:26.396731 2495 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskUci3DU/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781966829091300-388-0/minicluster-data/ts-2/wal/instance:
uuid: "7f1b44e3bb694105afe8822fc7cb348b"
format_stamp: "Formatted at 2026-06-20 14:47:26 on dist-test-slave-q9j6"
server_key: "4991f8785e8e1f2576f5283d21d7abd0"
server_key_iv: "d0062fae206815f0b3c69ae66e33dbf1"
server_key_version: "encryptionkey@0"
I20260620 14:47:26.402057 2495 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260620 14:47:26.406018 2510 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 14:47:26.407500 2495 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260620 14:47:26.407750 2495 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskUci3DU/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781966829091300-388-0/minicluster-data/ts-2/data,/tmp/dist-test-taskUci3DU/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781966829091300-388-0/minicluster-data/ts-2/wal
uuid: "7f1b44e3bb694105afe8822fc7cb348b"
format_stamp: "Formatted at 2026-06-20 14:47:26 on dist-test-slave-q9j6"
server_key: "4991f8785e8e1f2576f5283d21d7abd0"
server_key_iv: "d0062fae206815f0b3c69ae66e33dbf1"
server_key_version: "encryptionkey@0"
I20260620 14:47:26.407984 2495 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskUci3DU/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781966829091300-388-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskUci3DU/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781966829091300-388-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskUci3DU/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781966829091300-388-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 14:47:26.424962 2495 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 14:47:26.429499 2495 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 14:47:26.429795 2495 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 14:47:26.431025 2352 heartbeater.cc:503] Master 127.0.97.62:37975 was elected leader, sending a full tablet report...
I20260620 14:47:26.433030 2495 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 14:47:26.433148 2495 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 14:47:26.433277 2495 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 14:47:26.433360 2495 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Jun 20 14:47:26 dist-test-slave-q9j6 krb5kdc[2100](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781966846, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.3@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260620 14:47:26.459561 2240 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:47:26.433589 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.3:54935 (local address 127.0.97.62:37975)
0620 14:47:26.433799 (+ 210us) server_negotiation.cc:206] Beginning negotiation
0620 14:47:26.433826 (+ 27us) server_negotiation.cc:399] Waiting for connection header
0620 14:47:26.435551 (+ 1725us) server_negotiation.cc:407] Connection header received
0620 14:47:26.438077 (+ 2526us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:47:26.438084 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:47:26.438202 (+ 118us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:47:26.438338 (+ 136us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:47:26.441317 (+ 2979us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:47:26.442281 (+ 964us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:47:26.445731 (+ 3450us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:47:26.446139 (+ 408us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:47:26.446246 (+ 107us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:47:26.451534 (+ 5288us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 14:47:26.451544 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 14:47:26.451552 (+ 8us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 14:47:26.451632 (+ 80us) server_negotiation.cc:922] calling sasl_server_start()
0620 14:47:26.455881 (+ 4249us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:47:26.456995 (+ 1114us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:47:26.457005 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:47:26.457008 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:47:26.457215 (+ 207us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:47:26.457737 (+ 522us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:47:26.457742 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:47:26.457746 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:47:26.458031 (+ 285us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 14:47:26.458204 (+ 173us) server_negotiation.cc:1035] Waiting for connection context
0620 14:47:26.458973 (+ 769us) server_negotiation.cc:299] Negotiation successful
0620 14:47:26.459201 (+ 228us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":77}
I20260620 14:47:26.460805 2519 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:47:26.434405 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:37975 (local address 127.0.97.3:54935)
0620 14:47:26.435389 (+ 984us) negotiation.cc:107] Waiting for socket to connect
0620 14:47:26.435430 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0620 14:47:26.437664 (+ 2234us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 14:47:26.438843 (+ 1179us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 14:47:26.438865 (+ 22us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 14:47:26.439960 (+ 1095us) client_negotiation.cc:189] Negotiated authn=SASL
0620 14:47:26.441075 (+ 1115us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:47:26.441113 (+ 38us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:47:26.442492 (+ 1379us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:47:26.442503 (+ 11us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:47:26.445520 (+ 3017us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:47:26.445540 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:47:26.446424 (+ 884us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:47:26.446434 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:47:26.446807 (+ 373us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:47:26.447738 (+ 931us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 14:47:26.447780 (+ 42us) client_negotiation.cc:656] calling sasl_client_start()
0620 14:47:26.451198 (+ 3418us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 14:47:26.456080 (+ 4882us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 14:47:26.456093 (+ 13us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 14:47:26.456120 (+ 27us) client_negotiation.cc:760] calling sasl_client_step()
0620 14:47:26.456853 (+ 733us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 14:47:26.457372 (+ 519us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 14:47:26.457376 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 14:47:26.457378 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 14:47:26.457604 (+ 226us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 14:47:26.458173 (+ 569us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 14:47:26.458182 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 14:47:26.458560 (+ 378us) client_negotiation.cc:769] Sending connection context
0620 14:47:26.458841 (+ 281us) client_negotiation.cc:240] Negotiation successful
0620 14:47:26.459416 (+ 575us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":624,"thread_start_us":391,"threads_started":1}
I20260620 14:47:26.483376 2240 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:47:26.466982 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.3:53503 (local address 127.0.97.62:37975)
0620 14:47:26.467133 (+ 151us) server_negotiation.cc:206] Beginning negotiation
0620 14:47:26.467137 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 14:47:26.468140 (+ 1003us) server_negotiation.cc:407] Connection header received
0620 14:47:26.468340 (+ 200us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:47:26.468346 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:47:26.468427 (+ 81us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:47:26.468526 (+ 99us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:47:26.469753 (+ 1227us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:47:26.470671 (+ 918us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:47:26.473499 (+ 2828us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:47:26.473949 (+ 450us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:47:26.474100 (+ 151us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:47:26.476737 (+ 2637us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 14:47:26.476742 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 14:47:26.476746 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 14:47:26.476779 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0620 14:47:26.480346 (+ 3567us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:47:26.481020 (+ 674us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:47:26.481026 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:47:26.481029 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:47:26.481218 (+ 189us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:47:26.481681 (+ 463us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:47:26.481688 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:47:26.481694 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:47:26.482107 (+ 413us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 14:47:26.482331 (+ 224us) server_negotiation.cc:1035] Waiting for connection context
0620 14:47:26.482755 (+ 424us) server_negotiation.cc:299] Negotiation successful
0620 14:47:26.482976 (+ 221us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":41}
I20260620 14:47:26.483443 2590 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:47:26.466982 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:37975 (local address 127.0.97.3:53503)
0620 14:47:26.468035 (+ 1053us) negotiation.cc:107] Waiting for socket to connect
0620 14:47:26.468052 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0620 14:47:26.468221 (+ 169us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 14:47:26.468553 (+ 332us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 14:47:26.468557 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 14:47:26.469131 (+ 574us) client_negotiation.cc:189] Negotiated authn=SASL
0620 14:47:26.469603 (+ 472us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:47:26.469614 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:47:26.470865 (+ 1251us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:47:26.470870 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:47:26.473340 (+ 2470us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:47:26.473356 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:47:26.474213 (+ 857us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:47:26.474221 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:47:26.474371 (+ 150us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:47:26.475133 (+ 762us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 14:47:26.475178 (+ 45us) client_negotiation.cc:656] calling sasl_client_start()
0620 14:47:26.476515 (+ 1337us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 14:47:26.480487 (+ 3972us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 14:47:26.480493 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 14:47:26.480497 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0620 14:47:26.480892 (+ 395us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 14:47:26.481352 (+ 460us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 14:47:26.481357 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 14:47:26.481361 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0620 14:47:26.481527 (+ 166us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 14:47:26.482298 (+ 771us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 14:47:26.482303 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 14:47:26.482486 (+ 183us) client_negotiation.cc:769] Sending connection context
0620 14:47:26.482624 (+ 138us) client_negotiation.cc:240] Negotiation successful
0620 14:47:26.482980 (+ 356us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":907,"thread_start_us":507,"threads_started":1}
I20260620 14:47:26.488412 2495 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.3:32947
I20260620 14:47:26.488444 2629 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.3:32947 every 8 connection(s)
I20260620 14:47:26.489672 2495 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskUci3DU/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781966829091300-388-0/minicluster-data/ts-2/data/info.pb
I20260620 14:47:26.491626 2630 heartbeater.cc:347] Connected to a master server at 127.0.97.62:37975
I20260620 14:47:26.491873 2630 heartbeater.cc:464] Registering TS with master...
I20260620 14:47:26.492515 2630 heartbeater.cc:511] Master 127.0.97.62:37975 requested a full tablet report, sending...
I20260620 14:47:26.493868 2144 ts_manager.cc:194] Registered new tserver with Master: 7f1b44e3bb694105afe8822fc7cb348b (127.0.97.3:32947)
I20260620 14:47:26.494616 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskUci3DU/build/asan/bin/kudu as pid 2495
I20260620 14:47:26.494797 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskUci3DU/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781966829091300-388-0/minicluster-data/ts-2/wal/instance
I20260620 14:47:26.494789 2144 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.97.3@KRBTEST.COM'} at 127.0.97.3:54935
I20260620 14:47:26.495311 388 external_mini_cluster.cc:1428] Setting key 63bbd25274a4350f5cdf02170bfd81fa
I20260620 14:47:26.498184 388 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
Jun 20 14:47:26 dist-test-slave-q9j6 krb5kdc[2100](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781966846, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
Jun 20 14:47:26 dist-test-slave-q9j6 krb5kdc[2100](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781966846, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260620 14:47:26.544469 2240 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:47:26.531081 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40400 (local address 127.0.97.62:37975)
0620 14:47:26.531312 (+ 231us) server_negotiation.cc:206] Beginning negotiation
0620 14:47:26.531322 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0620 14:47:26.531545 (+ 223us) server_negotiation.cc:407] Connection header received
0620 14:47:26.531827 (+ 282us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:47:26.531832 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:47:26.531893 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:47:26.531996 (+ 103us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:47:26.533365 (+ 1369us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:47:26.534094 (+ 729us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:47:26.535321 (+ 1227us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:47:26.535778 (+ 457us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:47:26.535906 (+ 128us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:47:26.539699 (+ 3793us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 14:47:26.539704 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 14:47:26.539707 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 14:47:26.539739 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0620 14:47:26.541942 (+ 2203us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:47:26.542538 (+ 596us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:47:26.542543 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:47:26.542547 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:47:26.542668 (+ 121us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:47:26.543021 (+ 353us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:47:26.543026 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:47:26.543030 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:47:26.543296 (+ 266us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 14:47:26.543543 (+ 247us) server_negotiation.cc:1035] Waiting for connection context
0620 14:47:26.543881 (+ 338us) server_negotiation.cc:299] Negotiation successful
0620 14:47:26.544100 (+ 219us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":71}
I20260620 14:47:26.548540 2144 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:40400:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260620 14:47:26.555295 2144 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260620 14:47:26.579493 2646 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:47:26.568474 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46328 (local address 127.0.97.3:32947)
0620 14:47:26.572767 (+ 4293us) server_negotiation.cc:206] Beginning negotiation
0620 14:47:26.572773 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 14:47:26.572798 (+ 25us) server_negotiation.cc:407] Connection header received
0620 14:47:26.572869 (+ 71us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:47:26.572875 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:47:26.573055 (+ 180us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:47:26.573264 (+ 209us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 14:47:26.573926 (+ 662us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:47:26.575091 (+ 1165us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:47:26.577333 (+ 2242us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:47:26.578281 (+ 948us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:47:26.578432 (+ 151us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:47:26.578693 (+ 261us) server_negotiation.cc:1035] Waiting for connection context
0620 14:47:26.578964 (+ 271us) server_negotiation.cc:299] Negotiation successful
0620 14:47:26.579129 (+ 165us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":4161,"thread_start_us":3994,"threads_started":1}
I20260620 14:47:26.579619 2647 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:47:26.567892 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.3:32947 (local address 127.0.0.1:46328)
0620 14:47:26.570175 (+ 2283us) negotiation.cc:107] Waiting for socket to connect
0620 14:47:26.570200 (+ 25us) client_negotiation.cc:174] Beginning negotiation
0620 14:47:26.570359 (+ 159us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 14:47:26.573243 (+ 2884us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 14:47:26.573248 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 14:47:26.573275 (+ 27us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 14:47:26.573729 (+ 454us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:47:26.573740 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:47:26.575215 (+ 1475us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:47:26.575223 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:47:26.577182 (+ 1959us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:47:26.577195 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:47:26.578495 (+ 1300us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:47:26.578501 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:47:26.578710 (+ 209us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:47:26.578771 (+ 61us) client_negotiation.cc:769] Sending connection context
0620 14:47:26.578927 (+ 156us) client_negotiation.cc:240] Negotiation successful
0620 14:47:26.579126 (+ 199us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":2123,"mutex_wait_us":30,"thread_start_us":788,"threads_started":1}
I20260620 14:47:26.580510 2645 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:47:26.567592 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.2:37155 (local address 127.0.0.1:41730)
0620 14:47:26.568719 (+ 1127us) negotiation.cc:107] Waiting for socket to connect
0620 14:47:26.568768 (+ 49us) client_negotiation.cc:174] Beginning negotiation
0620 14:47:26.568949 (+ 181us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 14:47:26.570454 (+ 1505us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 14:47:26.570465 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 14:47:26.570506 (+ 41us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 14:47:26.570992 (+ 486us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:47:26.571013 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:47:26.573449 (+ 2436us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:47:26.573461 (+ 12us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:47:26.576790 (+ 3329us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:47:26.576812 (+ 22us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:47:26.579384 (+ 2572us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:47:26.579393 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:47:26.579687 (+ 294us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:47:26.579742 (+ 55us) client_negotiation.cc:769] Sending connection context
0620 14:47:26.579822 (+ 80us) client_negotiation.cc:240] Negotiation successful
0620 14:47:26.580017 (+ 195us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":793,"mutex_wait_us":4,"threads_started":1}
I20260620 14:47:26.581665 2644 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:47:26.567781 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41730 (local address 127.0.97.2:37155)
0620 14:47:26.569515 (+ 1734us) server_negotiation.cc:206] Beginning negotiation
0620 14:47:26.569525 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0620 14:47:26.569554 (+ 29us) server_negotiation.cc:407] Connection header received
0620 14:47:26.569666 (+ 112us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:47:26.569676 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:47:26.569945 (+ 269us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:47:26.570149 (+ 204us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 14:47:26.571215 (+ 1066us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:47:26.573229 (+ 2014us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:47:26.577107 (+ 3878us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:47:26.579073 (+ 1966us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:47:26.580797 (+ 1724us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:47:26.581106 (+ 309us) server_negotiation.cc:1035] Waiting for connection context
0620 14:47:26.581186 (+ 80us) server_negotiation.cc:299] Negotiation successful
0620 14:47:26.581387 (+ 201us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1574,"thread_start_us":1470,"threads_started":1}
I20260620 14:47:26.584352 2560 tablet_service.cc:1467] Processing CreateTablet for tablet f2f5ceda90ee40a9b777efa5d4309c1a (DEFAULT_TABLE table=test-table [id=58708c43b2174db18b083fae06190952]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 14:47:26.585350 2421 tablet_service.cc:1467] Processing CreateTablet for tablet f2f5ceda90ee40a9b777efa5d4309c1a (DEFAULT_TABLE table=test-table [id=58708c43b2174db18b083fae06190952]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 14:47:26.586064 2560 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f2f5ceda90ee40a9b777efa5d4309c1a. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 14:47:26.586620 2421 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f2f5ceda90ee40a9b777efa5d4309c1a. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 14:47:26.587821 2647 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:47:26.569547 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:35927 (local address 127.0.0.1:48790)
0620 14:47:26.579864 (+ 10317us) negotiation.cc:107] Waiting for socket to connect
0620 14:47:26.579883 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0620 14:47:26.580099 (+ 216us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 14:47:26.581051 (+ 952us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 14:47:26.581055 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 14:47:26.581082 (+ 27us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 14:47:26.581264 (+ 182us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:47:26.581274 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:47:26.583300 (+ 2026us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:47:26.583306 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:47:26.585132 (+ 1826us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:47:26.585162 (+ 30us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:47:26.586997 (+ 1835us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:47:26.587002 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:47:26.587198 (+ 196us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:47:26.587232 (+ 34us) client_negotiation.cc:769] Sending connection context
0620 14:47:26.587336 (+ 104us) client_negotiation.cc:240] Negotiation successful
0620 14:47:26.587484 (+ 148us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":10225,"thread_start_us":2115,"threads_started":1}
I20260620 14:47:26.587908 2648 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:47:26.570021 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48790 (local address 127.0.97.1:35927)
0620 14:47:26.572234 (+ 2213us) server_negotiation.cc:206] Beginning negotiation
0620 14:47:26.572247 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0620 14:47:26.579990 (+ 7743us) server_negotiation.cc:407] Connection header received
0620 14:47:26.580310 (+ 320us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:47:26.580319 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:47:26.580602 (+ 283us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:47:26.580783 (+ 181us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 14:47:26.581455 (+ 672us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:47:26.583094 (+ 1639us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:47:26.585350 (+ 2256us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:47:26.586812 (+ 1462us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:47:26.586939 (+ 127us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:47:26.587146 (+ 207us) server_negotiation.cc:1035] Waiting for connection context
0620 14:47:26.587397 (+ 251us) server_negotiation.cc:299] Negotiation successful
0620 14:47:26.587568 (+ 171us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":2042,"thread_start_us":1921,"threads_started":1}
I20260620 14:47:26.591934 2282 tablet_service.cc:1467] Processing CreateTablet for tablet f2f5ceda90ee40a9b777efa5d4309c1a (DEFAULT_TABLE table=test-table [id=58708c43b2174db18b083fae06190952]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 14:47:26.593438 2282 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f2f5ceda90ee40a9b777efa5d4309c1a. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 14:47:26.599987 2649 tablet_bootstrap.cc:492] T f2f5ceda90ee40a9b777efa5d4309c1a P e5717b124bc7489e8817304e3108a0eb: Bootstrap starting.
I20260620 14:47:26.603142 2649 tablet_bootstrap.cc:654] T f2f5ceda90ee40a9b777efa5d4309c1a P e5717b124bc7489e8817304e3108a0eb: Neither blocks nor log segments found. Creating new log.
I20260620 14:47:26.604111 2650 tablet_bootstrap.cc:492] T f2f5ceda90ee40a9b777efa5d4309c1a P 7f1b44e3bb694105afe8822fc7cb348b: Bootstrap starting.
I20260620 14:47:26.605649 2649 log.cc:824] T f2f5ceda90ee40a9b777efa5d4309c1a P e5717b124bc7489e8817304e3108a0eb: Log is configured to *not* fsync() on all Append() calls
I20260620 14:47:26.607867 2650 tablet_bootstrap.cc:654] T f2f5ceda90ee40a9b777efa5d4309c1a P 7f1b44e3bb694105afe8822fc7cb348b: Neither blocks nor log segments found. Creating new log.
I20260620 14:47:26.608896 2652 tablet_bootstrap.cc:492] T f2f5ceda90ee40a9b777efa5d4309c1a P 5aa58728f3b74083b017cdcd573bc626: Bootstrap starting.
I20260620 14:47:26.609000 2649 tablet_bootstrap.cc:492] T f2f5ceda90ee40a9b777efa5d4309c1a P e5717b124bc7489e8817304e3108a0eb: No bootstrap required, opened a new log
I20260620 14:47:26.609148 2650 log.cc:824] T f2f5ceda90ee40a9b777efa5d4309c1a P 7f1b44e3bb694105afe8822fc7cb348b: Log is configured to *not* fsync() on all Append() calls
I20260620 14:47:26.609404 2649 ts_tablet_manager.cc:1397] T f2f5ceda90ee40a9b777efa5d4309c1a P e5717b124bc7489e8817304e3108a0eb: Time spent bootstrapping tablet: real 0.010s user 0.005s sys 0.003s
I20260620 14:47:26.612432 2652 tablet_bootstrap.cc:654] T f2f5ceda90ee40a9b777efa5d4309c1a P 5aa58728f3b74083b017cdcd573bc626: Neither blocks nor log segments found. Creating new log.
I20260620 14:47:26.613878 2652 log.cc:824] T f2f5ceda90ee40a9b777efa5d4309c1a P 5aa58728f3b74083b017cdcd573bc626: Log is configured to *not* fsync() on all Append() calls
I20260620 14:47:26.614584 2650 tablet_bootstrap.cc:492] T f2f5ceda90ee40a9b777efa5d4309c1a P 7f1b44e3bb694105afe8822fc7cb348b: No bootstrap required, opened a new log
I20260620 14:47:26.615015 2650 ts_tablet_manager.cc:1397] T f2f5ceda90ee40a9b777efa5d4309c1a P 7f1b44e3bb694105afe8822fc7cb348b: Time spent bootstrapping tablet: real 0.011s user 0.007s sys 0.003s
I20260620 14:47:26.623997 2652 tablet_bootstrap.cc:492] T f2f5ceda90ee40a9b777efa5d4309c1a P 5aa58728f3b74083b017cdcd573bc626: No bootstrap required, opened a new log
I20260620 14:47:26.624029 2649 raft_consensus.cc:348] T f2f5ceda90ee40a9b777efa5d4309c1a P e5717b124bc7489e8817304e3108a0eb [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e5717b124bc7489e8817304e3108a0eb" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 37155 } } peers { permanent_uuid: "7f1b44e3bb694105afe8822fc7cb348b" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 32947 } } peers { permanent_uuid: "5aa58728f3b74083b017cdcd573bc626" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 35927 } }
I20260620 14:47:26.624341 2652 ts_tablet_manager.cc:1397] T f2f5ceda90ee40a9b777efa5d4309c1a P 5aa58728f3b74083b017cdcd573bc626: Time spent bootstrapping tablet: real 0.016s user 0.007s sys 0.006s
I20260620 14:47:26.624485 2649 raft_consensus.cc:374] T f2f5ceda90ee40a9b777efa5d4309c1a P e5717b124bc7489e8817304e3108a0eb [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 14:47:26.624615 2649 raft_consensus.cc:729] T f2f5ceda90ee40a9b777efa5d4309c1a P e5717b124bc7489e8817304e3108a0eb [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e5717b124bc7489e8817304e3108a0eb, State: Initialized, Role: FOLLOWER
I20260620 14:47:26.625259 2649 consensus_queue.cc:260] T f2f5ceda90ee40a9b777efa5d4309c1a P e5717b124bc7489e8817304e3108a0eb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e5717b124bc7489e8817304e3108a0eb" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 37155 } } peers { permanent_uuid: "7f1b44e3bb694105afe8822fc7cb348b" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 32947 } } peers { permanent_uuid: "5aa58728f3b74083b017cdcd573bc626" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 35927 } }
I20260620 14:47:26.626524 2491 heartbeater.cc:503] Master 127.0.97.62:37975 was elected leader, sending a full tablet report...
I20260620 14:47:26.627140 2649 ts_tablet_manager.cc:1428] T f2f5ceda90ee40a9b777efa5d4309c1a P e5717b124bc7489e8817304e3108a0eb: Time spent starting tablet: real 0.018s user 0.011s sys 0.003s
I20260620 14:47:26.627269 2650 raft_consensus.cc:348] T f2f5ceda90ee40a9b777efa5d4309c1a P 7f1b44e3bb694105afe8822fc7cb348b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e5717b124bc7489e8817304e3108a0eb" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 37155 } } peers { permanent_uuid: "7f1b44e3bb694105afe8822fc7cb348b" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 32947 } } peers { permanent_uuid: "5aa58728f3b74083b017cdcd573bc626" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 35927 } }
I20260620 14:47:26.627893 2650 raft_consensus.cc:374] T f2f5ceda90ee40a9b777efa5d4309c1a P 7f1b44e3bb694105afe8822fc7cb348b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 14:47:26.628050 2650 raft_consensus.cc:729] T f2f5ceda90ee40a9b777efa5d4309c1a P 7f1b44e3bb694105afe8822fc7cb348b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7f1b44e3bb694105afe8822fc7cb348b, State: Initialized, Role: FOLLOWER
I20260620 14:47:26.628677 2650 consensus_queue.cc:260] T f2f5ceda90ee40a9b777efa5d4309c1a P 7f1b44e3bb694105afe8822fc7cb348b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e5717b124bc7489e8817304e3108a0eb" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 37155 } } peers { permanent_uuid: "7f1b44e3bb694105afe8822fc7cb348b" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 32947 } } peers { permanent_uuid: "5aa58728f3b74083b017cdcd573bc626" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 35927 } }
I20260620 14:47:26.629889 2630 heartbeater.cc:503] Master 127.0.97.62:37975 was elected leader, sending a full tablet report...
I20260620 14:47:26.630661 2650 ts_tablet_manager.cc:1428] T f2f5ceda90ee40a9b777efa5d4309c1a P 7f1b44e3bb694105afe8822fc7cb348b: Time spent starting tablet: real 0.015s user 0.014s sys 0.000s
I20260620 14:47:26.636370 2652 raft_consensus.cc:348] T f2f5ceda90ee40a9b777efa5d4309c1a P 5aa58728f3b74083b017cdcd573bc626 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e5717b124bc7489e8817304e3108a0eb" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 37155 } } peers { permanent_uuid: "7f1b44e3bb694105afe8822fc7cb348b" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 32947 } } peers { permanent_uuid: "5aa58728f3b74083b017cdcd573bc626" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 35927 } }
I20260620 14:47:26.636757 2652 raft_consensus.cc:374] T f2f5ceda90ee40a9b777efa5d4309c1a P 5aa58728f3b74083b017cdcd573bc626 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 14:47:26.636855 2652 raft_consensus.cc:729] T f2f5ceda90ee40a9b777efa5d4309c1a P 5aa58728f3b74083b017cdcd573bc626 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5aa58728f3b74083b017cdcd573bc626, State: Initialized, Role: FOLLOWER
I20260620 14:47:26.637650 2652 consensus_queue.cc:260] T f2f5ceda90ee40a9b777efa5d4309c1a P 5aa58728f3b74083b017cdcd573bc626 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e5717b124bc7489e8817304e3108a0eb" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 37155 } } peers { permanent_uuid: "7f1b44e3bb694105afe8822fc7cb348b" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 32947 } } peers { permanent_uuid: "5aa58728f3b74083b017cdcd573bc626" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 35927 } }
I20260620 14:47:26.639665 2652 ts_tablet_manager.cc:1428] T f2f5ceda90ee40a9b777efa5d4309c1a P 5aa58728f3b74083b017cdcd573bc626: Time spent starting tablet: real 0.015s user 0.013s sys 0.000s
W20260620 14:47:26.673669 2353 tablet.cc:2369] T f2f5ceda90ee40a9b777efa5d4309c1a P 5aa58728f3b74083b017cdcd573bc626: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260620 14:47:26.694460 2655 raft_consensus.cc:482] T f2f5ceda90ee40a9b777efa5d4309c1a P e5717b124bc7489e8817304e3108a0eb [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260620 14:47:26.694741 2655 raft_consensus.cc:504] T f2f5ceda90ee40a9b777efa5d4309c1a P e5717b124bc7489e8817304e3108a0eb [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e5717b124bc7489e8817304e3108a0eb" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 37155 } } peers { permanent_uuid: "7f1b44e3bb694105afe8822fc7cb348b" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 32947 } } peers { permanent_uuid: "5aa58728f3b74083b017cdcd573bc626" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 35927 } }
I20260620 14:47:26.696488 2655 leader_election.cc:288] T f2f5ceda90ee40a9b777efa5d4309c1a P e5717b124bc7489e8817304e3108a0eb [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 7f1b44e3bb694105afe8822fc7cb348b (127.0.97.3:32947), 5aa58728f3b74083b017cdcd573bc626 (127.0.97.1:35927)
W20260620 14:47:26.702450 2492 tablet.cc:2369] T f2f5ceda90ee40a9b777efa5d4309c1a P e5717b124bc7489e8817304e3108a0eb: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260620 14:47:26.704012 2646 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:47:26.696925 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.2:58563 (local address 127.0.97.3:32947)
0620 14:47:26.697132 (+ 207us) server_negotiation.cc:206] Beginning negotiation
0620 14:47:26.697136 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 14:47:26.697693 (+ 557us) server_negotiation.cc:407] Connection header received
0620 14:47:26.697907 (+ 214us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:47:26.697914 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:47:26.697987 (+ 73us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:47:26.698090 (+ 103us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 14:47:26.698865 (+ 775us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:47:26.699884 (+ 1019us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:47:26.701754 (+ 1870us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:47:26.702979 (+ 1225us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:47:26.703102 (+ 123us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:47:26.703227 (+ 125us) server_negotiation.cc:1035] Waiting for connection context
0620 14:47:26.703588 (+ 361us) server_negotiation.cc:299] Negotiation successful
0620 14:47:26.703735 (+ 147us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":58}
I20260620 14:47:26.704100 2648 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:47:26.697980 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.2:45815 (local address 127.0.97.1:35927)
0620 14:47:26.698176 (+ 196us) server_negotiation.cc:206] Beginning negotiation
0620 14:47:26.698200 (+ 24us) server_negotiation.cc:399] Waiting for connection header
0620 14:47:26.698436 (+ 236us) server_negotiation.cc:407] Connection header received
0620 14:47:26.698558 (+ 122us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:47:26.698564 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:47:26.698657 (+ 93us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:47:26.698764 (+ 107us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 14:47:26.699420 (+ 656us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:47:26.700271 (+ 851us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:47:26.702093 (+ 1822us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:47:26.703107 (+ 1014us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:47:26.703237 (+ 130us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:47:26.703351 (+ 114us) server_negotiation.cc:1035] Waiting for connection context
0620 14:47:26.703728 (+ 377us) server_negotiation.cc:299] Negotiation successful
0620 14:47:26.703856 (+ 128us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":71}
I20260620 14:47:26.704231 2659 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:47:26.697749 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:35927 (local address 127.0.97.2:45815)
0620 14:47:26.698332 (+ 583us) negotiation.cc:107] Waiting for socket to connect
0620 14:47:26.698346 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0620 14:47:26.698457 (+ 111us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 14:47:26.698840 (+ 383us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 14:47:26.698845 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 14:47:26.698871 (+ 26us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 14:47:26.699261 (+ 390us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:47:26.699272 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:47:26.700400 (+ 1128us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:47:26.700403 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:47:26.701937 (+ 1534us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:47:26.701953 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:47:26.703407 (+ 1454us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:47:26.703413 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:47:26.703543 (+ 130us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:47:26.703565 (+ 22us) client_negotiation.cc:769] Sending connection context
0620 14:47:26.703691 (+ 126us) client_negotiation.cc:240] Negotiation successful
0620 14:47:26.703822 (+ 131us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":474,"thread_start_us":281,"threads_started":1}
I20260620 14:47:26.704543 2658 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:47:26.696858 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.3:32947 (local address 127.0.97.2:58563)
0620 14:47:26.697581 (+ 723us) negotiation.cc:107] Waiting for socket to connect
0620 14:47:26.697602 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0620 14:47:26.697741 (+ 139us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 14:47:26.698289 (+ 548us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 14:47:26.698293 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 14:47:26.698316 (+ 23us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 14:47:26.698686 (+ 370us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:47:26.698701 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:47:26.700002 (+ 1301us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:47:26.700008 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:47:26.701601 (+ 1593us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:47:26.701612 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:47:26.703224 (+ 1612us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:47:26.703231 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:47:26.703421 (+ 190us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:47:26.703451 (+ 30us) client_negotiation.cc:769] Sending connection context
0620 14:47:26.703563 (+ 112us) client_negotiation.cc:240] Negotiation successful
0620 14:47:26.703719 (+ 156us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":597,"mutex_wait_us":37,"thread_start_us":421,"threads_started":1}
I20260620 14:47:26.705483 2302 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f2f5ceda90ee40a9b777efa5d4309c1a" candidate_uuid: "e5717b124bc7489e8817304e3108a0eb" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5aa58728f3b74083b017cdcd573bc626" is_pre_election: true
I20260620 14:47:26.705483 2580 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f2f5ceda90ee40a9b777efa5d4309c1a" candidate_uuid: "e5717b124bc7489e8817304e3108a0eb" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7f1b44e3bb694105afe8822fc7cb348b" is_pre_election: true
I20260620 14:47:26.706066 2302 raft_consensus.cc:2454] T f2f5ceda90ee40a9b777efa5d4309c1a P 5aa58728f3b74083b017cdcd573bc626 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e5717b124bc7489e8817304e3108a0eb in term 0.
I20260620 14:47:26.706066 2580 raft_consensus.cc:2454] T f2f5ceda90ee40a9b777efa5d4309c1a P 7f1b44e3bb694105afe8822fc7cb348b [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e5717b124bc7489e8817304e3108a0eb in term 0.
I20260620 14:47:26.706753 2373 leader_election.cc:302] T f2f5ceda90ee40a9b777efa5d4309c1a P e5717b124bc7489e8817304e3108a0eb [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 7f1b44e3bb694105afe8822fc7cb348b, e5717b124bc7489e8817304e3108a0eb; no voters:
I20260620 14:47:26.707100 2655 raft_consensus.cc:2781] T f2f5ceda90ee40a9b777efa5d4309c1a P e5717b124bc7489e8817304e3108a0eb [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260620 14:47:26.707269 2655 raft_consensus.cc:482] T f2f5ceda90ee40a9b777efa5d4309c1a P e5717b124bc7489e8817304e3108a0eb [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260620 14:47:26.707386 2655 raft_consensus.cc:3037] T f2f5ceda90ee40a9b777efa5d4309c1a P e5717b124bc7489e8817304e3108a0eb [term 0 FOLLOWER]: Advancing to term 1
I20260620 14:47:26.709618 2655 raft_consensus.cc:504] T f2f5ceda90ee40a9b777efa5d4309c1a P e5717b124bc7489e8817304e3108a0eb [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e5717b124bc7489e8817304e3108a0eb" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 37155 } } peers { permanent_uuid: "7f1b44e3bb694105afe8822fc7cb348b" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 32947 } } peers { permanent_uuid: "5aa58728f3b74083b017cdcd573bc626" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 35927 } }
I20260620 14:47:26.710234 2655 leader_election.cc:288] T f2f5ceda90ee40a9b777efa5d4309c1a P e5717b124bc7489e8817304e3108a0eb [CANDIDATE]: Term 1 election: Requested vote from peers 7f1b44e3bb694105afe8822fc7cb348b (127.0.97.3:32947), 5aa58728f3b74083b017cdcd573bc626 (127.0.97.1:35927)
I20260620 14:47:26.710763 2580 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f2f5ceda90ee40a9b777efa5d4309c1a" candidate_uuid: "e5717b124bc7489e8817304e3108a0eb" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7f1b44e3bb694105afe8822fc7cb348b"
I20260620 14:47:26.710857 2302 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f2f5ceda90ee40a9b777efa5d4309c1a" candidate_uuid: "e5717b124bc7489e8817304e3108a0eb" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5aa58728f3b74083b017cdcd573bc626"
I20260620 14:47:26.710994 2580 raft_consensus.cc:3037] T f2f5ceda90ee40a9b777efa5d4309c1a P 7f1b44e3bb694105afe8822fc7cb348b [term 0 FOLLOWER]: Advancing to term 1
I20260620 14:47:26.711092 2302 raft_consensus.cc:3037] T f2f5ceda90ee40a9b777efa5d4309c1a P 5aa58728f3b74083b017cdcd573bc626 [term 0 FOLLOWER]: Advancing to term 1
I20260620 14:47:26.713548 2302 raft_consensus.cc:2454] T f2f5ceda90ee40a9b777efa5d4309c1a P 5aa58728f3b74083b017cdcd573bc626 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e5717b124bc7489e8817304e3108a0eb in term 1.
I20260620 14:47:26.713567 2580 raft_consensus.cc:2454] T f2f5ceda90ee40a9b777efa5d4309c1a P 7f1b44e3bb694105afe8822fc7cb348b [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e5717b124bc7489e8817304e3108a0eb in term 1.
I20260620 14:47:26.713968 2373 leader_election.cc:302] T f2f5ceda90ee40a9b777efa5d4309c1a P e5717b124bc7489e8817304e3108a0eb [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5aa58728f3b74083b017cdcd573bc626, e5717b124bc7489e8817304e3108a0eb; no voters:
I20260620 14:47:26.714316 2655 raft_consensus.cc:2781] T f2f5ceda90ee40a9b777efa5d4309c1a P e5717b124bc7489e8817304e3108a0eb [term 1 FOLLOWER]: Leader election won for term 1
I20260620 14:47:26.714771 2655 raft_consensus.cc:686] T f2f5ceda90ee40a9b777efa5d4309c1a P e5717b124bc7489e8817304e3108a0eb [term 1 LEADER]: Becoming Leader. State: Replica: e5717b124bc7489e8817304e3108a0eb, State: Running, Role: LEADER
I20260620 14:47:26.715315 2655 consensus_queue.cc:237] T f2f5ceda90ee40a9b777efa5d4309c1a P e5717b124bc7489e8817304e3108a0eb [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e5717b124bc7489e8817304e3108a0eb" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 37155 } } peers { permanent_uuid: "7f1b44e3bb694105afe8822fc7cb348b" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 32947 } } peers { permanent_uuid: "5aa58728f3b74083b017cdcd573bc626" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 35927 } }
I20260620 14:47:26.720551 2145 catalog_manager.cc:5495] T f2f5ceda90ee40a9b777efa5d4309c1a P e5717b124bc7489e8817304e3108a0eb reported cstate change: term changed from 0 to 1, leader changed from <none> to e5717b124bc7489e8817304e3108a0eb (127.0.97.2). New cstate: current_term: 1 leader_uuid: "e5717b124bc7489e8817304e3108a0eb" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e5717b124bc7489e8817304e3108a0eb" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 37155 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "7f1b44e3bb694105afe8822fc7cb348b" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 32947 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "5aa58728f3b74083b017cdcd573bc626" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 35927 } health_report { overall_health: UNKNOWN } } }
W20260620 14:47:26.741394 2631 tablet.cc:2369] T f2f5ceda90ee40a9b777efa5d4309c1a P 7f1b44e3bb694105afe8822fc7cb348b: Can't schedule compaction. Clean time has not been advanced past its initial value.
Jun 20 14:47:26 dist-test-slave-q9j6 krb5kdc[2100](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781966844, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.62@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260620 14:47:26.784940 2240 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:47:26.770438 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40414 (local address 127.0.97.62:37975)
0620 14:47:26.770610 (+ 172us) server_negotiation.cc:206] Beginning negotiation
0620 14:47:26.770615 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 14:47:26.771211 (+ 596us) server_negotiation.cc:407] Connection header received
0620 14:47:26.771416 (+ 205us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:47:26.771421 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:47:26.771482 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:47:26.771580 (+ 98us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:47:26.772865 (+ 1285us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:47:26.773608 (+ 743us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:47:26.774944 (+ 1336us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:47:26.775508 (+ 564us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:47:26.775661 (+ 153us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:47:26.779541 (+ 3880us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 14:47:26.779546 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 14:47:26.779549 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 14:47:26.779577 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0620 14:47:26.782174 (+ 2597us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:47:26.782914 (+ 740us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:47:26.782922 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:47:26.782926 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:47:26.783066 (+ 140us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:47:26.783407 (+ 341us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:47:26.783411 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:47:26.783415 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:47:26.783747 (+ 332us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 14:47:26.783889 (+ 142us) server_negotiation.cc:1035] Waiting for connection context
0620 14:47:26.784297 (+ 408us) server_negotiation.cc:299] Negotiation successful
0620 14:47:26.784615 (+ 318us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":138,"server-negotiator.queue_time_us":57}
I20260620 14:47:26.784976 2667 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:47:26.770466 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:37975 (local address 127.0.0.1:40414)
0620 14:47:26.771096 (+ 630us) negotiation.cc:107] Waiting for socket to connect
0620 14:47:26.771116 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0620 14:47:26.771289 (+ 173us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 14:47:26.771654 (+ 365us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 14:47:26.771658 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 14:47:26.772402 (+ 744us) client_negotiation.cc:189] Negotiated authn=SASL
0620 14:47:26.772695 (+ 293us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:47:26.772705 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:47:26.773765 (+ 1060us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:47:26.773770 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:47:26.774800 (+ 1030us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:47:26.774812 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:47:26.775753 (+ 941us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:47:26.775760 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:47:26.775924 (+ 164us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:47:26.776778 (+ 854us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 14:47:26.776806 (+ 28us) client_negotiation.cc:656] calling sasl_client_start()
0620 14:47:26.779348 (+ 2542us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 14:47:26.782322 (+ 2974us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 14:47:26.782335 (+ 13us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 14:47:26.782348 (+ 13us) client_negotiation.cc:760] calling sasl_client_step()
0620 14:47:26.782799 (+ 451us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 14:47:26.783177 (+ 378us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 14:47:26.783182 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 14:47:26.783185 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0620 14:47:26.783302 (+ 117us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 14:47:26.783872 (+ 570us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 14:47:26.783880 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 14:47:26.784041 (+ 161us) client_negotiation.cc:769] Sending connection context
0620 14:47:26.784214 (+ 173us) client_negotiation.cc:240] Negotiation successful
0620 14:47:26.784508 (+ 294us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":512,"thread_start_us":370,"threads_started":1}
I20260620 14:47:26.789801 2145 catalog_manager.cc:1908] Servicing CreateTable request from {username='kudu', principal='kudu/127.0.97.62@KRBTEST.COM'} at 127.0.0.1:40414:
name: "kudu_system.kudu_transactions"
schema {
columns {
name: "txn_id"
type: INT64
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "entry_type"
type: INT8
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "identifier"
type: STRING
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "metadata"
type: STRING
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
rows: "<redacted>""\006\001\000\000\000\000\000\000\000\000\007\001@B\017\000\000\000\000\000"
indirect_data: "<redacted>"""
}
partition_schema {
range_schema {
columns {
name: "txn_id"
}
}
}
table_type: TXN_STATUS_TABLE
W20260620 14:47:26.790845 2145 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table kudu_system.kudu_transactions in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260620 14:47:26.797472 2282 tablet_service.cc:1467] Processing CreateTablet for tablet 4b9ec36214934a8e9a673fceb97bac8f (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=49b69d45675f4d5cb61c7ea80454f21a]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260620 14:47:26.797669 2421 tablet_service.cc:1467] Processing CreateTablet for tablet 4b9ec36214934a8e9a673fceb97bac8f (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=49b69d45675f4d5cb61c7ea80454f21a]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260620 14:47:26.797916 2560 tablet_service.cc:1467] Processing CreateTablet for tablet 4b9ec36214934a8e9a673fceb97bac8f (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=49b69d45675f4d5cb61c7ea80454f21a]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260620 14:47:26.798511 2282 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4b9ec36214934a8e9a673fceb97bac8f. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 14:47:26.798602 2421 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4b9ec36214934a8e9a673fceb97bac8f. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 14:47:26.798651 2560 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4b9ec36214934a8e9a673fceb97bac8f. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 14:47:26.804687 2652 tablet_bootstrap.cc:492] T 4b9ec36214934a8e9a673fceb97bac8f P 5aa58728f3b74083b017cdcd573bc626: Bootstrap starting.
I20260620 14:47:26.806057 2649 tablet_bootstrap.cc:492] T 4b9ec36214934a8e9a673fceb97bac8f P e5717b124bc7489e8817304e3108a0eb: Bootstrap starting.
I20260620 14:47:26.806083 2652 tablet_bootstrap.cc:654] T 4b9ec36214934a8e9a673fceb97bac8f P 5aa58728f3b74083b017cdcd573bc626: Neither blocks nor log segments found. Creating new log.
I20260620 14:47:26.807397 2650 tablet_bootstrap.cc:492] T 4b9ec36214934a8e9a673fceb97bac8f P 7f1b44e3bb694105afe8822fc7cb348b: Bootstrap starting.
I20260620 14:47:26.807853 2649 tablet_bootstrap.cc:654] T 4b9ec36214934a8e9a673fceb97bac8f P e5717b124bc7489e8817304e3108a0eb: Neither blocks nor log segments found. Creating new log.
I20260620 14:47:26.808849 2652 tablet_bootstrap.cc:492] T 4b9ec36214934a8e9a673fceb97bac8f P 5aa58728f3b74083b017cdcd573bc626: No bootstrap required, opened a new log
I20260620 14:47:26.808923 2650 tablet_bootstrap.cc:654] T 4b9ec36214934a8e9a673fceb97bac8f P 7f1b44e3bb694105afe8822fc7cb348b: Neither blocks nor log segments found. Creating new log.
I20260620 14:47:26.808982 2652 ts_tablet_manager.cc:1397] T 4b9ec36214934a8e9a673fceb97bac8f P 5aa58728f3b74083b017cdcd573bc626: Time spent bootstrapping tablet: real 0.004s user 0.003s sys 0.000s
I20260620 14:47:26.809616 2652 raft_consensus.cc:348] T 4b9ec36214934a8e9a673fceb97bac8f P 5aa58728f3b74083b017cdcd573bc626 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5aa58728f3b74083b017cdcd573bc626" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 35927 } } peers { permanent_uuid: "e5717b124bc7489e8817304e3108a0eb" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 37155 } } peers { permanent_uuid: "7f1b44e3bb694105afe8822fc7cb348b" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 32947 } }
I20260620 14:47:26.809903 2652 raft_consensus.cc:374] T 4b9ec36214934a8e9a673fceb97bac8f P 5aa58728f3b74083b017cdcd573bc626 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 14:47:26.810026 2652 raft_consensus.cc:729] T 4b9ec36214934a8e9a673fceb97bac8f P 5aa58728f3b74083b017cdcd573bc626 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5aa58728f3b74083b017cdcd573bc626, State: Initialized, Role: FOLLOWER
I20260620 14:47:26.810392 2649 tablet_bootstrap.cc:492] T 4b9ec36214934a8e9a673fceb97bac8f P e5717b124bc7489e8817304e3108a0eb: No bootstrap required, opened a new log
I20260620 14:47:26.810329 2652 consensus_queue.cc:260] T 4b9ec36214934a8e9a673fceb97bac8f P 5aa58728f3b74083b017cdcd573bc626 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5aa58728f3b74083b017cdcd573bc626" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 35927 } } peers { permanent_uuid: "e5717b124bc7489e8817304e3108a0eb" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 37155 } } peers { permanent_uuid: "7f1b44e3bb694105afe8822fc7cb348b" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 32947 } }
I20260620 14:47:26.810575 2649 ts_tablet_manager.cc:1397] T 4b9ec36214934a8e9a673fceb97bac8f P e5717b124bc7489e8817304e3108a0eb: Time spent bootstrapping tablet: real 0.005s user 0.001s sys 0.003s
I20260620 14:47:26.811004 2652 ts_tablet_manager.cc:1428] T 4b9ec36214934a8e9a673fceb97bac8f P 5aa58728f3b74083b017cdcd573bc626: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20260620 14:47:26.811126 2650 tablet_bootstrap.cc:492] T 4b9ec36214934a8e9a673fceb97bac8f P 7f1b44e3bb694105afe8822fc7cb348b: No bootstrap required, opened a new log
I20260620 14:47:26.810915 2657 tablet_replica.cc:439] T 4b9ec36214934a8e9a673fceb97bac8f P 5aa58728f3b74083b017cdcd573bc626: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5aa58728f3b74083b017cdcd573bc626" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 35927 } } peers { permanent_uuid: "e5717b124bc7489e8817304e3108a0eb" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 37155 } } peers { permanent_uuid: "7f1b44e3bb694105afe8822fc7cb348b" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 32947 } } }
I20260620 14:47:26.811277 2650 ts_tablet_manager.cc:1397] T 4b9ec36214934a8e9a673fceb97bac8f P 7f1b44e3bb694105afe8822fc7cb348b: Time spent bootstrapping tablet: real 0.004s user 0.003s sys 0.000s
I20260620 14:47:26.811391 2657 tablet_replica.cc:442] T 4b9ec36214934a8e9a673fceb97bac8f P 5aa58728f3b74083b017cdcd573bc626: This TxnStatusTablet replica's current role is: FOLLOWER
I20260620 14:47:26.812036 2649 raft_consensus.cc:348] T 4b9ec36214934a8e9a673fceb97bac8f P e5717b124bc7489e8817304e3108a0eb [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5aa58728f3b74083b017cdcd573bc626" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 35927 } } peers { permanent_uuid: "e5717b124bc7489e8817304e3108a0eb" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 37155 } } peers { permanent_uuid: "7f1b44e3bb694105afe8822fc7cb348b" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 32947 } }
I20260620 14:47:26.812068 2650 raft_consensus.cc:348] T 4b9ec36214934a8e9a673fceb97bac8f P 7f1b44e3bb694105afe8822fc7cb348b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5aa58728f3b74083b017cdcd573bc626" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 35927 } } peers { permanent_uuid: "e5717b124bc7489e8817304e3108a0eb" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 37155 } } peers { permanent_uuid: "7f1b44e3bb694105afe8822fc7cb348b" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 32947 } }
I20260620 14:47:26.812294 2649 raft_consensus.cc:374] T 4b9ec36214934a8e9a673fceb97bac8f P e5717b124bc7489e8817304e3108a0eb [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 14:47:26.812316 2650 raft_consensus.cc:374] T 4b9ec36214934a8e9a673fceb97bac8f P 7f1b44e3bb694105afe8822fc7cb348b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 14:47:26.812476 2649 raft_consensus.cc:729] T 4b9ec36214934a8e9a673fceb97bac8f P e5717b124bc7489e8817304e3108a0eb [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e5717b124bc7489e8817304e3108a0eb, State: Initialized, Role: FOLLOWER
I20260620 14:47:26.812474 2650 raft_consensus.cc:729] T 4b9ec36214934a8e9a673fceb97bac8f P 7f1b44e3bb694105afe8822fc7cb348b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7f1b44e3bb694105afe8822fc7cb348b, State: Initialized, Role: FOLLOWER
I20260620 14:47:26.812742 2649 consensus_queue.cc:260] T 4b9ec36214934a8e9a673fceb97bac8f P e5717b124bc7489e8817304e3108a0eb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5aa58728f3b74083b017cdcd573bc626" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 35927 } } peers { permanent_uuid: "e5717b124bc7489e8817304e3108a0eb" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 37155 } } peers { permanent_uuid: "7f1b44e3bb694105afe8822fc7cb348b" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 32947 } }
I20260620 14:47:26.812721 2650 consensus_queue.cc:260] T 4b9ec36214934a8e9a673fceb97bac8f P 7f1b44e3bb694105afe8822fc7cb348b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5aa58728f3b74083b017cdcd573bc626" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 35927 } } peers { permanent_uuid: "e5717b124bc7489e8817304e3108a0eb" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 37155 } } peers { permanent_uuid: "7f1b44e3bb694105afe8822fc7cb348b" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 32947 } }
I20260620 14:47:26.813131 2655 tablet_replica.cc:439] T 4b9ec36214934a8e9a673fceb97bac8f P e5717b124bc7489e8817304e3108a0eb: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5aa58728f3b74083b017cdcd573bc626" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 35927 } } peers { permanent_uuid: "e5717b124bc7489e8817304e3108a0eb" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 37155 } } peers { permanent_uuid: "7f1b44e3bb694105afe8822fc7cb348b" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 32947 } } }
I20260620 14:47:26.813342 2650 ts_tablet_manager.cc:1428] T 4b9ec36214934a8e9a673fceb97bac8f P 7f1b44e3bb694105afe8822fc7cb348b: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20260620 14:47:26.813408 2655 tablet_replica.cc:442] T 4b9ec36214934a8e9a673fceb97bac8f P e5717b124bc7489e8817304e3108a0eb: This TxnStatusTablet replica's current role is: FOLLOWER
I20260620 14:47:26.813186 2656 tablet_replica.cc:439] T 4b9ec36214934a8e9a673fceb97bac8f P 7f1b44e3bb694105afe8822fc7cb348b: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5aa58728f3b74083b017cdcd573bc626" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 35927 } } peers { permanent_uuid: "e5717b124bc7489e8817304e3108a0eb" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 37155 } } peers { permanent_uuid: "7f1b44e3bb694105afe8822fc7cb348b" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 32947 } } }
I20260620 14:47:26.813547 2656 tablet_replica.cc:442] T 4b9ec36214934a8e9a673fceb97bac8f P 7f1b44e3bb694105afe8822fc7cb348b: This TxnStatusTablet replica's current role is: FOLLOWER
I20260620 14:47:26.813937 2649 ts_tablet_manager.cc:1428] T 4b9ec36214934a8e9a673fceb97bac8f P e5717b124bc7489e8817304e3108a0eb: Time spent starting tablet: real 0.003s user 0.002s sys 0.000s
I20260620 14:47:26.857264 2656 raft_consensus.cc:482] T 4b9ec36214934a8e9a673fceb97bac8f P 7f1b44e3bb694105afe8822fc7cb348b [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260620 14:47:26.857508 2656 raft_consensus.cc:504] T 4b9ec36214934a8e9a673fceb97bac8f P 7f1b44e3bb694105afe8822fc7cb348b [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5aa58728f3b74083b017cdcd573bc626" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 35927 } } peers { permanent_uuid: "e5717b124bc7489e8817304e3108a0eb" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 37155 } } peers { permanent_uuid: "7f1b44e3bb694105afe8822fc7cb348b" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 32947 } }
I20260620 14:47:26.858999 2656 leader_election.cc:288] T 4b9ec36214934a8e9a673fceb97bac8f P 7f1b44e3bb694105afe8822fc7cb348b [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 5aa58728f3b74083b017cdcd573bc626 (127.0.97.1:35927), e5717b124bc7489e8817304e3108a0eb (127.0.97.2:37155)
I20260620 14:47:26.865815 2648 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:47:26.859556 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.3:49857 (local address 127.0.97.1:35927)
0620 14:47:26.859772 (+ 216us) server_negotiation.cc:206] Beginning negotiation
0620 14:47:26.859780 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0620 14:47:26.859800 (+ 20us) server_negotiation.cc:407] Connection header received
0620 14:47:26.859871 (+ 71us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:47:26.859876 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:47:26.859970 (+ 94us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:47:26.860075 (+ 105us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 14:47:26.860726 (+ 651us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:47:26.861749 (+ 1023us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:47:26.863916 (+ 2167us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:47:26.864924 (+ 1008us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:47:26.865070 (+ 146us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:47:26.865209 (+ 139us) server_negotiation.cc:1035] Waiting for connection context
0620 14:47:26.865467 (+ 258us) server_negotiation.cc:299] Negotiation successful
0620 14:47:26.865568 (+ 101us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":94}
I20260620 14:47:26.865841 2519 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:47:26.859357 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:35927 (local address 127.0.97.3:49857)
0620 14:47:26.859537 (+ 180us) negotiation.cc:107] Waiting for socket to connect
0620 14:47:26.859558 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0620 14:47:26.859719 (+ 161us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 14:47:26.860141 (+ 422us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 14:47:26.860148 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 14:47:26.860188 (+ 40us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 14:47:26.860520 (+ 332us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:47:26.860533 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:47:26.861876 (+ 1343us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:47:26.861879 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:47:26.863717 (+ 1838us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:47:26.863732 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:47:26.865078 (+ 1346us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:47:26.865083 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:47:26.865275 (+ 192us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:47:26.865311 (+ 36us) client_negotiation.cc:769] Sending connection context
0620 14:47:26.865411 (+ 100us) client_negotiation.cc:240] Negotiation successful
0620 14:47:26.865550 (+ 139us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":42}
I20260620 14:47:26.866714 2644 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:47:26.859457 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.3:58369 (local address 127.0.97.2:37155)
0620 14:47:26.859683 (+ 226us) server_negotiation.cc:206] Beginning negotiation
0620 14:47:26.859689 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 14:47:26.860702 (+ 1013us) server_negotiation.cc:407] Connection header received
0620 14:47:26.860826 (+ 124us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:47:26.860832 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:47:26.860903 (+ 71us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:47:26.860988 (+ 85us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 14:47:26.861451 (+ 463us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:47:26.862294 (+ 843us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:47:26.864532 (+ 2238us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:47:26.865692 (+ 1160us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:47:26.865843 (+ 151us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:47:26.865964 (+ 121us) server_negotiation.cc:1035] Waiting for connection context
0620 14:47:26.866375 (+ 411us) server_negotiation.cc:299] Negotiation successful
0620 14:47:26.866503 (+ 128us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":75}
I20260620 14:47:26.866712 2671 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:47:26.859372 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.2:37155 (local address 127.0.97.3:58369)
0620 14:47:26.860593 (+ 1221us) negotiation.cc:107] Waiting for socket to connect
0620 14:47:26.860612 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0620 14:47:26.860721 (+ 109us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 14:47:26.861013 (+ 292us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 14:47:26.861016 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 14:47:26.861032 (+ 16us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 14:47:26.861302 (+ 270us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:47:26.861309 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:47:26.862494 (+ 1185us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:47:26.862499 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:47:26.864379 (+ 1880us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:47:26.864394 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:47:26.865951 (+ 1557us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:47:26.865957 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:47:26.866135 (+ 178us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:47:26.866171 (+ 36us) client_negotiation.cc:769] Sending connection context
0620 14:47:26.866268 (+ 97us) client_negotiation.cc:240] Negotiation successful
0620 14:47:26.866373 (+ 105us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1084,"mutex_wait_us":444,"thread_start_us":375,"threads_started":1}
I20260620 14:47:26.866735 2302 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4b9ec36214934a8e9a673fceb97bac8f" candidate_uuid: "7f1b44e3bb694105afe8822fc7cb348b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5aa58728f3b74083b017cdcd573bc626" is_pre_election: true
I20260620 14:47:26.866989 2302 raft_consensus.cc:2454] T 4b9ec36214934a8e9a673fceb97bac8f P 5aa58728f3b74083b017cdcd573bc626 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7f1b44e3bb694105afe8822fc7cb348b in term 0.
I20260620 14:47:26.867465 2512 leader_election.cc:302] T 4b9ec36214934a8e9a673fceb97bac8f P 7f1b44e3bb694105afe8822fc7cb348b [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5aa58728f3b74083b017cdcd573bc626, 7f1b44e3bb694105afe8822fc7cb348b; no voters:
I20260620 14:47:26.867552 2441 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4b9ec36214934a8e9a673fceb97bac8f" candidate_uuid: "7f1b44e3bb694105afe8822fc7cb348b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e5717b124bc7489e8817304e3108a0eb" is_pre_election: true
I20260620 14:47:26.867894 2656 raft_consensus.cc:2781] T 4b9ec36214934a8e9a673fceb97bac8f P 7f1b44e3bb694105afe8822fc7cb348b [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260620 14:47:26.868013 2656 raft_consensus.cc:482] T 4b9ec36214934a8e9a673fceb97bac8f P 7f1b44e3bb694105afe8822fc7cb348b [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260620 14:47:26.868120 2656 raft_consensus.cc:3037] T 4b9ec36214934a8e9a673fceb97bac8f P 7f1b44e3bb694105afe8822fc7cb348b [term 0 FOLLOWER]: Advancing to term 1
I20260620 14:47:26.868041 2441 raft_consensus.cc:2454] T 4b9ec36214934a8e9a673fceb97bac8f P e5717b124bc7489e8817304e3108a0eb [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7f1b44e3bb694105afe8822fc7cb348b in term 0.
I20260620 14:47:26.870371 2656 raft_consensus.cc:504] T 4b9ec36214934a8e9a673fceb97bac8f P 7f1b44e3bb694105afe8822fc7cb348b [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5aa58728f3b74083b017cdcd573bc626" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 35927 } } peers { permanent_uuid: "e5717b124bc7489e8817304e3108a0eb" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 37155 } } peers { permanent_uuid: "7f1b44e3bb694105afe8822fc7cb348b" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 32947 } }
I20260620 14:47:26.870880 2656 leader_election.cc:288] T 4b9ec36214934a8e9a673fceb97bac8f P 7f1b44e3bb694105afe8822fc7cb348b [CANDIDATE]: Term 1 election: Requested vote from peers 5aa58728f3b74083b017cdcd573bc626 (127.0.97.1:35927), e5717b124bc7489e8817304e3108a0eb (127.0.97.2:37155)
I20260620 14:47:26.871306 2302 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4b9ec36214934a8e9a673fceb97bac8f" candidate_uuid: "7f1b44e3bb694105afe8822fc7cb348b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5aa58728f3b74083b017cdcd573bc626"
I20260620 14:47:26.871392 2441 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4b9ec36214934a8e9a673fceb97bac8f" candidate_uuid: "7f1b44e3bb694105afe8822fc7cb348b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e5717b124bc7489e8817304e3108a0eb"
I20260620 14:47:26.871532 2302 raft_consensus.cc:3037] T 4b9ec36214934a8e9a673fceb97bac8f P 5aa58728f3b74083b017cdcd573bc626 [term 0 FOLLOWER]: Advancing to term 1
I20260620 14:47:26.871558 2441 raft_consensus.cc:3037] T 4b9ec36214934a8e9a673fceb97bac8f P e5717b124bc7489e8817304e3108a0eb [term 0 FOLLOWER]: Advancing to term 1
I20260620 14:47:26.874511 2302 raft_consensus.cc:2454] T 4b9ec36214934a8e9a673fceb97bac8f P 5aa58728f3b74083b017cdcd573bc626 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7f1b44e3bb694105afe8822fc7cb348b in term 1.
I20260620 14:47:26.874524 2441 raft_consensus.cc:2454] T 4b9ec36214934a8e9a673fceb97bac8f P e5717b124bc7489e8817304e3108a0eb [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7f1b44e3bb694105afe8822fc7cb348b in term 1.
I20260620 14:47:26.874935 2512 leader_election.cc:302] T 4b9ec36214934a8e9a673fceb97bac8f P 7f1b44e3bb694105afe8822fc7cb348b [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5aa58728f3b74083b017cdcd573bc626, 7f1b44e3bb694105afe8822fc7cb348b; no voters:
I20260620 14:47:26.875267 2656 raft_consensus.cc:2781] T 4b9ec36214934a8e9a673fceb97bac8f P 7f1b44e3bb694105afe8822fc7cb348b [term 1 FOLLOWER]: Leader election won for term 1
I20260620 14:47:26.875787 2656 raft_consensus.cc:686] T 4b9ec36214934a8e9a673fceb97bac8f P 7f1b44e3bb694105afe8822fc7cb348b [term 1 LEADER]: Becoming Leader. State: Replica: 7f1b44e3bb694105afe8822fc7cb348b, State: Running, Role: LEADER
I20260620 14:47:26.876343 2656 consensus_queue.cc:237] T 4b9ec36214934a8e9a673fceb97bac8f P 7f1b44e3bb694105afe8822fc7cb348b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5aa58728f3b74083b017cdcd573bc626" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 35927 } } peers { permanent_uuid: "e5717b124bc7489e8817304e3108a0eb" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 37155 } } peers { permanent_uuid: "7f1b44e3bb694105afe8822fc7cb348b" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 32947 } }
I20260620 14:47:26.879755 2672 tablet_replica.cc:439] T 4b9ec36214934a8e9a673fceb97bac8f P 7f1b44e3bb694105afe8822fc7cb348b: TxnStatusTablet state changed. Reason: New leader 7f1b44e3bb694105afe8822fc7cb348b. Latest consensus state: current_term: 1 leader_uuid: "7f1b44e3bb694105afe8822fc7cb348b" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5aa58728f3b74083b017cdcd573bc626" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 35927 } } peers { permanent_uuid: "e5717b124bc7489e8817304e3108a0eb" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 37155 } } peers { permanent_uuid: "7f1b44e3bb694105afe8822fc7cb348b" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 32947 } } }
I20260620 14:47:26.880178 2672 tablet_replica.cc:442] T 4b9ec36214934a8e9a673fceb97bac8f P 7f1b44e3bb694105afe8822fc7cb348b: This TxnStatusTablet replica's current role is: LEADER
I20260620 14:47:26.880924 2674 txn_status_manager.cc:873] Waiting until node catch up with all replicated operations in previous term...
I20260620 14:47:26.881218 2674 txn_status_manager.cc:929] Loading transaction status metadata into memory...
I20260620 14:47:26.881278 2143 catalog_manager.cc:5495] T 4b9ec36214934a8e9a673fceb97bac8f P 7f1b44e3bb694105afe8822fc7cb348b reported cstate change: term changed from 0 to 1, leader changed from <none> to 7f1b44e3bb694105afe8822fc7cb348b (127.0.97.3). New cstate: current_term: 1 leader_uuid: "7f1b44e3bb694105afe8822fc7cb348b" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5aa58728f3b74083b017cdcd573bc626" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 35927 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "e5717b124bc7489e8817304e3108a0eb" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 37155 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "7f1b44e3bb694105afe8822fc7cb348b" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 32947 } health_report { overall_health: HEALTHY } } }
I20260620 14:47:26.914485 2667 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:47:26.908726 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.3:32947 (local address 127.0.0.1:46336)
0620 14:47:26.908991 (+ 265us) negotiation.cc:107] Waiting for socket to connect
0620 14:47:26.909009 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0620 14:47:26.909171 (+ 162us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 14:47:26.909633 (+ 462us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 14:47:26.909638 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 14:47:26.909649 (+ 11us) client_negotiation.cc:189] Negotiated authn=TOKEN
0620 14:47:26.909840 (+ 191us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:47:26.909849 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:47:26.910929 (+ 1080us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:47:26.910935 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:47:26.912105 (+ 1170us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:47:26.912117 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:47:26.912871 (+ 754us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:47:26.912877 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:47:26.913070 (+ 193us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:47:26.913124 (+ 54us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0620 14:47:26.913954 (+ 830us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0620 14:47:26.913974 (+ 20us) client_negotiation.cc:769] Sending connection context
0620 14:47:26.914057 (+ 83us) client_negotiation.cc:240] Negotiation successful
0620 14:47:26.914202 (+ 145us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":66}
I20260620 14:47:26.914544 2646 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:47:26.908679 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46336 (local address 127.0.97.3:32947)
0620 14:47:26.908852 (+ 173us) server_negotiation.cc:206] Beginning negotiation
0620 14:47:26.908861 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0620 14:47:26.909080 (+ 219us) server_negotiation.cc:407] Connection header received
0620 14:47:26.909388 (+ 308us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:47:26.909392 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:47:26.909469 (+ 77us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:47:26.909614 (+ 145us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 14:47:26.909967 (+ 353us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:47:26.910801 (+ 834us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:47:26.912241 (+ 1440us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:47:26.912717 (+ 476us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:47:26.912857 (+ 140us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:47:26.913311 (+ 454us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 14:47:26.913847 (+ 536us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0620 14:47:26.913994 (+ 147us) server_negotiation.cc:1035] Waiting for connection context
0620 14:47:26.914074 (+ 80us) server_negotiation.cc:299] Negotiation successful
0620 14:47:26.914155 (+ 81us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":49}
I20260620 14:47:26.923326 2302 raft_consensus.cc:1261] T 4b9ec36214934a8e9a673fceb97bac8f P 5aa58728f3b74083b017cdcd573bc626 [term 1 FOLLOWER]: Refusing update from remote peer 7f1b44e3bb694105afe8822fc7cb348b: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 14:47:26.923370 2441 raft_consensus.cc:1261] T 4b9ec36214934a8e9a673fceb97bac8f P e5717b124bc7489e8817304e3108a0eb [term 1 FOLLOWER]: Refusing update from remote peer 7f1b44e3bb694105afe8822fc7cb348b: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 14:47:26.924388 2672 consensus_queue.cc:1035] T 4b9ec36214934a8e9a673fceb97bac8f P 7f1b44e3bb694105afe8822fc7cb348b [LEADER]: Connected to new peer: Peer: permanent_uuid: "5aa58728f3b74083b017cdcd573bc626" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 35927 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 14:47:26.924760 2656 consensus_queue.cc:1035] T 4b9ec36214934a8e9a673fceb97bac8f P 7f1b44e3bb694105afe8822fc7cb348b [LEADER]: Connected to new peer: Peer: permanent_uuid: "e5717b124bc7489e8817304e3108a0eb" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 37155 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 14:47:26.941581 2655 tablet_replica.cc:439] T 4b9ec36214934a8e9a673fceb97bac8f P e5717b124bc7489e8817304e3108a0eb: TxnStatusTablet state changed. Reason: New leader 7f1b44e3bb694105afe8822fc7cb348b. Latest consensus state: current_term: 1 leader_uuid: "7f1b44e3bb694105afe8822fc7cb348b" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5aa58728f3b74083b017cdcd573bc626" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 35927 } } peers { permanent_uuid: "e5717b124bc7489e8817304e3108a0eb" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 37155 } } peers { permanent_uuid: "7f1b44e3bb694105afe8822fc7cb348b" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 32947 } } }
I20260620 14:47:26.941927 2655 tablet_replica.cc:442] T 4b9ec36214934a8e9a673fceb97bac8f P e5717b124bc7489e8817304e3108a0eb: This TxnStatusTablet replica's current role is: FOLLOWER
I20260620 14:47:26.944872 2657 tablet_replica.cc:439] T 4b9ec36214934a8e9a673fceb97bac8f P 5aa58728f3b74083b017cdcd573bc626: TxnStatusTablet state changed. Reason: New leader 7f1b44e3bb694105afe8822fc7cb348b. Latest consensus state: current_term: 1 leader_uuid: "7f1b44e3bb694105afe8822fc7cb348b" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5aa58728f3b74083b017cdcd573bc626" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 35927 } } peers { permanent_uuid: "e5717b124bc7489e8817304e3108a0eb" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 37155 } } peers { permanent_uuid: "7f1b44e3bb694105afe8822fc7cb348b" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 32947 } } }
I20260620 14:47:26.945178 2657 tablet_replica.cc:442] T 4b9ec36214934a8e9a673fceb97bac8f P 5aa58728f3b74083b017cdcd573bc626: This TxnStatusTablet replica's current role is: FOLLOWER
I20260620 14:47:26.946117 2677 mvcc.cc:205] Tried to move back new op lower bound from 7298936204988153856 to 7298936204809441280. Current Snapshot: MvccSnapshot[applied={T|T < 7298936204988153856}]
I20260620 14:47:26.952081 2656 tablet_replica.cc:439] T 4b9ec36214934a8e9a673fceb97bac8f P 7f1b44e3bb694105afe8822fc7cb348b: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "7f1b44e3bb694105afe8822fc7cb348b" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5aa58728f3b74083b017cdcd573bc626" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 35927 } } peers { permanent_uuid: "e5717b124bc7489e8817304e3108a0eb" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 37155 } } peers { permanent_uuid: "7f1b44e3bb694105afe8822fc7cb348b" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 32947 } } }
I20260620 14:47:26.952381 2656 tablet_replica.cc:442] T 4b9ec36214934a8e9a673fceb97bac8f P 7f1b44e3bb694105afe8822fc7cb348b: This TxnStatusTablet replica's current role is: LEADER
I20260620 14:47:26.952661 2656 tablet_replica.cc:439] T 4b9ec36214934a8e9a673fceb97bac8f P 7f1b44e3bb694105afe8822fc7cb348b: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "7f1b44e3bb694105afe8822fc7cb348b" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5aa58728f3b74083b017cdcd573bc626" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 35927 } } peers { permanent_uuid: "e5717b124bc7489e8817304e3108a0eb" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 37155 } } peers { permanent_uuid: "7f1b44e3bb694105afe8822fc7cb348b" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 32947 } } }
I20260620 14:47:26.952855 2656 tablet_replica.cc:442] T 4b9ec36214934a8e9a673fceb97bac8f P 7f1b44e3bb694105afe8822fc7cb348b: This TxnStatusTablet replica's current role is: LEADER
I20260620 14:47:26.977523 2655 tablet_replica.cc:439] T 4b9ec36214934a8e9a673fceb97bac8f P e5717b124bc7489e8817304e3108a0eb: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "7f1b44e3bb694105afe8822fc7cb348b" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5aa58728f3b74083b017cdcd573bc626" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 35927 } } peers { permanent_uuid: "e5717b124bc7489e8817304e3108a0eb" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 37155 } } peers { permanent_uuid: "7f1b44e3bb694105afe8822fc7cb348b" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 32947 } } }
I20260620 14:47:26.977838 2655 tablet_replica.cc:442] T 4b9ec36214934a8e9a673fceb97bac8f P e5717b124bc7489e8817304e3108a0eb: This TxnStatusTablet replica's current role is: FOLLOWER
I20260620 14:47:26.980060 2657 tablet_replica.cc:439] T 4b9ec36214934a8e9a673fceb97bac8f P 5aa58728f3b74083b017cdcd573bc626: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "7f1b44e3bb694105afe8822fc7cb348b" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5aa58728f3b74083b017cdcd573bc626" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 35927 } } peers { permanent_uuid: "e5717b124bc7489e8817304e3108a0eb" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 37155 } } peers { permanent_uuid: "7f1b44e3bb694105afe8822fc7cb348b" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 32947 } } }
I20260620 14:47:26.980316 2657 tablet_replica.cc:442] T 4b9ec36214934a8e9a673fceb97bac8f P 5aa58728f3b74083b017cdcd573bc626: This TxnStatusTablet replica's current role is: FOLLOWER
I20260620 14:47:26.983578 2644 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:47:26.971352 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41746 (local address 127.0.97.2:37155)
0620 14:47:26.971659 (+ 307us) server_negotiation.cc:206] Beginning negotiation
0620 14:47:26.971668 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0620 14:47:26.971692 (+ 24us) server_negotiation.cc:407] Connection header received
0620 14:47:26.971832 (+ 140us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:47:26.971841 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:47:26.972063 (+ 222us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:47:26.972252 (+ 189us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 14:47:26.973173 (+ 921us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:47:26.974742 (+ 1569us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:47:26.977794 (+ 3052us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:47:26.978726 (+ 932us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:47:26.978889 (+ 163us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:47:26.979709 (+ 820us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 14:47:26.982807 (+ 3098us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0620 14:47:26.982959 (+ 152us) server_negotiation.cc:1035] Waiting for connection context
0620 14:47:26.983186 (+ 227us) server_negotiation.cc:299] Negotiation successful
0620 14:47:26.983284 (+ 98us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":103}
I20260620 14:47:27.000976 2646 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:47:26.995264 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.2:60315 (local address 127.0.97.3:32947)
0620 14:47:26.995428 (+ 164us) server_negotiation.cc:206] Beginning negotiation
0620 14:47:26.995432 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 14:47:26.996115 (+ 683us) server_negotiation.cc:407] Connection header received
0620 14:47:26.996339 (+ 224us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:47:26.996349 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:47:26.996419 (+ 70us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:47:26.996515 (+ 96us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 14:47:26.997061 (+ 546us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:47:26.997760 (+ 699us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:47:26.999113 (+ 1353us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:47:26.999565 (+ 452us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:47:26.999718 (+ 153us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:47:27.000221 (+ 503us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 14:47:27.000335 (+ 114us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0620 14:47:27.000431 (+ 96us) server_negotiation.cc:1035] Waiting for connection context
0620 14:47:27.000564 (+ 133us) server_negotiation.cc:299] Negotiation successful
0620 14:47:27.000649 (+ 85us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":44}
I20260620 14:47:27.001081 2688 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:47:26.995262 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.3:32947 (local address 127.0.97.2:60315)
0620 14:47:26.996016 (+ 754us) negotiation.cc:107] Waiting for socket to connect
0620 14:47:26.996033 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0620 14:47:26.996194 (+ 161us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 14:47:26.996527 (+ 333us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 14:47:26.996531 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 14:47:26.996542 (+ 11us) client_negotiation.cc:189] Negotiated authn=TOKEN
0620 14:47:26.996889 (+ 347us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:47:26.996898 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:47:26.997899 (+ 1001us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:47:26.997905 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:47:26.998971 (+ 1066us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:47:26.998983 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:47:26.999801 (+ 818us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:47:26.999807 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:47:26.999935 (+ 128us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:47:26.999991 (+ 56us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0620 14:47:27.000434 (+ 443us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0620 14:47:27.000448 (+ 14us) client_negotiation.cc:769] Sending connection context
0620 14:47:27.000544 (+ 96us) client_negotiation.cc:240] Negotiation successful
0620 14:47:27.000683 (+ 139us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":567,"thread_start_us":387,"threads_started":1}
I20260620 14:47:27.011545 2302 raft_consensus.cc:1261] T f2f5ceda90ee40a9b777efa5d4309c1a P 5aa58728f3b74083b017cdcd573bc626 [term 1 FOLLOWER]: Refusing update from remote peer e5717b124bc7489e8817304e3108a0eb: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 14:47:27.012315 2580 raft_consensus.cc:1261] T f2f5ceda90ee40a9b777efa5d4309c1a P 7f1b44e3bb694105afe8822fc7cb348b [term 1 FOLLOWER]: Refusing update from remote peer e5717b124bc7489e8817304e3108a0eb: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 14:47:27.012228 2660 consensus_queue.cc:1035] T f2f5ceda90ee40a9b777efa5d4309c1a P e5717b124bc7489e8817304e3108a0eb [LEADER]: Connected to new peer: Peer: permanent_uuid: "5aa58728f3b74083b017cdcd573bc626" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 35927 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 14:47:27.012979 2660 consensus_queue.cc:1035] T f2f5ceda90ee40a9b777efa5d4309c1a P e5717b124bc7489e8817304e3108a0eb [LEADER]: Connected to new peer: Peer: permanent_uuid: "7f1b44e3bb694105afe8822fc7cb348b" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 32947 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 14:47:27.050966 2644 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:47:27.044935 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.3:43527 (local address 127.0.97.2:37155)
0620 14:47:27.045130 (+ 195us) server_negotiation.cc:206] Beginning negotiation
0620 14:47:27.045137 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0620 14:47:27.045728 (+ 591us) server_negotiation.cc:407] Connection header received
0620 14:47:27.045921 (+ 193us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:47:27.045927 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:47:27.046002 (+ 75us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:47:27.046104 (+ 102us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 14:47:27.046732 (+ 628us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:47:27.047571 (+ 839us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:47:27.049070 (+ 1499us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:47:27.049488 (+ 418us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:47:27.049616 (+ 128us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:47:27.050172 (+ 556us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 14:47:27.050276 (+ 104us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0620 14:47:27.050371 (+ 95us) server_negotiation.cc:1035] Waiting for connection context
0620 14:47:27.050559 (+ 188us) server_negotiation.cc:299] Negotiation successful
0620 14:47:27.050651 (+ 92us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":48}
I20260620 14:47:27.051102 2694 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:47:27.044869 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.2:37155 (local address 127.0.97.3:43527)
0620 14:47:27.045615 (+ 746us) negotiation.cc:107] Waiting for socket to connect
0620 14:47:27.045640 (+ 25us) client_negotiation.cc:174] Beginning negotiation
0620 14:47:27.045785 (+ 145us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 14:47:27.046117 (+ 332us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 14:47:27.046122 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 14:47:27.046134 (+ 12us) client_negotiation.cc:189] Negotiated authn=TOKEN
0620 14:47:27.046566 (+ 432us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:47:27.046580 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:47:27.047734 (+ 1154us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:47:27.047739 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:47:27.048915 (+ 1176us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:47:27.048927 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:47:27.049686 (+ 759us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:47:27.049692 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:47:27.049906 (+ 214us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:47:27.049986 (+ 80us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0620 14:47:27.050372 (+ 386us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0620 14:47:27.050384 (+ 12us) client_negotiation.cc:769] Sending connection context
0620 14:47:27.050510 (+ 126us) client_negotiation.cc:240] Negotiation successful
0620 14:47:27.050674 (+ 164us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":535,"thread_start_us":484,"threads_started":1}
I20260620 14:47:27.093195 2144 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:40400:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 14:47:27.093506 2144 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:40400:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 14:47:27.097736 2144 catalog_manager.cc:5782] T 00000000000000000000000000000000 P dbf70d79ae0b41f28a6637ed9ab0ecc4: Sending DeleteTablet for 3 replicas of tablet f2f5ceda90ee40a9b777efa5d4309c1a
I20260620 14:47:27.098942 2421 tablet_service.cc:1514] Processing DeleteTablet for tablet f2f5ceda90ee40a9b777efa5d4309c1a with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 14:47:27 UTC) from {username='kudu', principal='kudu/127.0.97.62@KRBTEST.COM'} at 127.0.0.1:41730
I20260620 14:47:27.098958 2560 tablet_service.cc:1514] Processing DeleteTablet for tablet f2f5ceda90ee40a9b777efa5d4309c1a with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 14:47:27 UTC) from {username='kudu', principal='kudu/127.0.97.62@KRBTEST.COM'} at 127.0.0.1:46328
I20260620 14:47:27.100200 2282 tablet_service.cc:1514] Processing DeleteTablet for tablet f2f5ceda90ee40a9b777efa5d4309c1a with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 14:47:27 UTC) from {username='kudu', principal='kudu/127.0.97.62@KRBTEST.COM'} at 127.0.0.1:48790
I20260620 14:47:27.108032 2143 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:40400:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260620 14:47:27.108803 2143 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260620 14:47:27.117669 2281 tablet_service.cc:1467] Processing CreateTablet for tablet 70245e00c5ed4751a2f7d04dc7dac140 (DEFAULT_TABLE table=test-table [id=243167de760e48f382fe3b25335c6a85]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 14:47:27.117856 2559 tablet_service.cc:1467] Processing CreateTablet for tablet 70245e00c5ed4751a2f7d04dc7dac140 (DEFAULT_TABLE table=test-table [id=243167de760e48f382fe3b25335c6a85]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 14:47:27.118358 2281 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 70245e00c5ed4751a2f7d04dc7dac140. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 14:47:27.118373 2559 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 70245e00c5ed4751a2f7d04dc7dac140. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 14:47:27.125087 2420 tablet_service.cc:1467] Processing CreateTablet for tablet 70245e00c5ed4751a2f7d04dc7dac140 (DEFAULT_TABLE table=test-table [id=243167de760e48f382fe3b25335c6a85]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 14:47:27.125660 2420 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 70245e00c5ed4751a2f7d04dc7dac140. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 14:47:27.127076 2650 tablet_bootstrap.cc:492] T 70245e00c5ed4751a2f7d04dc7dac140 P 7f1b44e3bb694105afe8822fc7cb348b: Bootstrap starting.
I20260620 14:47:27.129099 2650 tablet_bootstrap.cc:654] T 70245e00c5ed4751a2f7d04dc7dac140 P 7f1b44e3bb694105afe8822fc7cb348b: Neither blocks nor log segments found. Creating new log.
I20260620 14:47:27.129184 2699 tablet_replica.cc:330] T f2f5ceda90ee40a9b777efa5d4309c1a P 5aa58728f3b74083b017cdcd573bc626: stopping tablet replica
I20260620 14:47:27.129684 2699 raft_consensus.cc:2229] T f2f5ceda90ee40a9b777efa5d4309c1a P 5aa58728f3b74083b017cdcd573bc626 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 14:47:27.130103 2699 raft_consensus.cc:2258] T f2f5ceda90ee40a9b777efa5d4309c1a P 5aa58728f3b74083b017cdcd573bc626 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 14:47:27.131161 2697 tablet_replica.cc:330] T f2f5ceda90ee40a9b777efa5d4309c1a P 7f1b44e3bb694105afe8822fc7cb348b: stopping tablet replica
I20260620 14:47:27.131752 2697 raft_consensus.cc:2229] T f2f5ceda90ee40a9b777efa5d4309c1a P 7f1b44e3bb694105afe8822fc7cb348b [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 14:47:27.132148 2697 raft_consensus.cc:2258] T f2f5ceda90ee40a9b777efa5d4309c1a P 7f1b44e3bb694105afe8822fc7cb348b [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 14:47:27.132956 2652 tablet_bootstrap.cc:492] T 70245e00c5ed4751a2f7d04dc7dac140 P 5aa58728f3b74083b017cdcd573bc626: Bootstrap starting.
I20260620 14:47:27.133839 2649 tablet_bootstrap.cc:492] T 70245e00c5ed4751a2f7d04dc7dac140 P e5717b124bc7489e8817304e3108a0eb: Bootstrap starting.
I20260620 14:47:27.134660 2652 tablet_bootstrap.cc:654] T 70245e00c5ed4751a2f7d04dc7dac140 P 5aa58728f3b74083b017cdcd573bc626: Neither blocks nor log segments found. Creating new log.
I20260620 14:47:27.134630 2697 ts_tablet_manager.cc:1905] T f2f5ceda90ee40a9b777efa5d4309c1a P 7f1b44e3bb694105afe8822fc7cb348b: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 14:47:27.135342 2698 tablet_replica.cc:330] T f2f5ceda90ee40a9b777efa5d4309c1a P e5717b124bc7489e8817304e3108a0eb: stopping tablet replica
I20260620 14:47:27.135715 2649 tablet_bootstrap.cc:654] T 70245e00c5ed4751a2f7d04dc7dac140 P e5717b124bc7489e8817304e3108a0eb: Neither blocks nor log segments found. Creating new log.
I20260620 14:47:27.136027 2698 raft_consensus.cc:2229] T f2f5ceda90ee40a9b777efa5d4309c1a P e5717b124bc7489e8817304e3108a0eb [term 1 LEADER]: Raft consensus shutting down.
I20260620 14:47:27.136812 2698 raft_consensus.cc:2258] T f2f5ceda90ee40a9b777efa5d4309c1a P e5717b124bc7489e8817304e3108a0eb [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 14:47:27.139940 2650 tablet_bootstrap.cc:492] T 70245e00c5ed4751a2f7d04dc7dac140 P 7f1b44e3bb694105afe8822fc7cb348b: No bootstrap required, opened a new log
I20260620 14:47:27.140125 2650 ts_tablet_manager.cc:1397] T 70245e00c5ed4751a2f7d04dc7dac140 P 7f1b44e3bb694105afe8822fc7cb348b: Time spent bootstrapping tablet: real 0.013s user 0.006s sys 0.000s
I20260620 14:47:27.141023 2698 ts_tablet_manager.cc:1905] T f2f5ceda90ee40a9b777efa5d4309c1a P e5717b124bc7489e8817304e3108a0eb: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 14:47:27.140957 2650 raft_consensus.cc:348] T 70245e00c5ed4751a2f7d04dc7dac140 P 7f1b44e3bb694105afe8822fc7cb348b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5aa58728f3b74083b017cdcd573bc626" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 35927 } } peers { permanent_uuid: "e5717b124bc7489e8817304e3108a0eb" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 37155 } } peers { permanent_uuid: "7f1b44e3bb694105afe8822fc7cb348b" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 32947 } }
I20260620 14:47:27.141217 2650 raft_consensus.cc:374] T 70245e00c5ed4751a2f7d04dc7dac140 P 7f1b44e3bb694105afe8822fc7cb348b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 14:47:27.141304 2650 raft_consensus.cc:729] T 70245e00c5ed4751a2f7d04dc7dac140 P 7f1b44e3bb694105afe8822fc7cb348b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7f1b44e3bb694105afe8822fc7cb348b, State: Initialized, Role: FOLLOWER
I20260620 14:47:27.141571 2650 consensus_queue.cc:260] T 70245e00c5ed4751a2f7d04dc7dac140 P 7f1b44e3bb694105afe8822fc7cb348b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5aa58728f3b74083b017cdcd573bc626" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 35927 } } peers { permanent_uuid: "e5717b124bc7489e8817304e3108a0eb" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 37155 } } peers { permanent_uuid: "7f1b44e3bb694105afe8822fc7cb348b" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 32947 } }
I20260620 14:47:27.142114 2650 ts_tablet_manager.cc:1428] T 70245e00c5ed4751a2f7d04dc7dac140 P 7f1b44e3bb694105afe8822fc7cb348b: Time spent starting tablet: real 0.002s user 0.000s sys 0.000s
I20260620 14:47:27.143312 2697 ts_tablet_manager.cc:1918] T f2f5ceda90ee40a9b777efa5d4309c1a P 7f1b44e3bb694105afe8822fc7cb348b: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260620 14:47:27.143476 2697 log.cc:1196] T f2f5ceda90ee40a9b777efa5d4309c1a P 7f1b44e3bb694105afe8822fc7cb348b: Deleting WAL directory at /tmp/dist-test-taskUci3DU/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781966829091300-388-0/minicluster-data/ts-2/wal/wals/f2f5ceda90ee40a9b777efa5d4309c1a
I20260620 14:47:27.144055 2697 ts_tablet_manager.cc:1939] T f2f5ceda90ee40a9b777efa5d4309c1a P 7f1b44e3bb694105afe8822fc7cb348b: Deleting consensus metadata
I20260620 14:47:27.145792 2130 catalog_manager.cc:4841] TS 7f1b44e3bb694105afe8822fc7cb348b (127.0.97.3:32947): tablet f2f5ceda90ee40a9b777efa5d4309c1a (table test-table [id=58708c43b2174db18b083fae06190952]) successfully deleted
I20260620 14:47:27.148175 2699 ts_tablet_manager.cc:1905] T f2f5ceda90ee40a9b777efa5d4309c1a P 5aa58728f3b74083b017cdcd573bc626: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 14:47:27.148977 2652 tablet_bootstrap.cc:492] T 70245e00c5ed4751a2f7d04dc7dac140 P 5aa58728f3b74083b017cdcd573bc626: No bootstrap required, opened a new log
I20260620 14:47:27.149155 2652 ts_tablet_manager.cc:1397] T 70245e00c5ed4751a2f7d04dc7dac140 P 5aa58728f3b74083b017cdcd573bc626: Time spent bootstrapping tablet: real 0.016s user 0.003s sys 0.004s
I20260620 14:47:27.149955 2652 raft_consensus.cc:348] T 70245e00c5ed4751a2f7d04dc7dac140 P 5aa58728f3b74083b017cdcd573bc626 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5aa58728f3b74083b017cdcd573bc626" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 35927 } } peers { permanent_uuid: "e5717b124bc7489e8817304e3108a0eb" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 37155 } } peers { permanent_uuid: "7f1b44e3bb694105afe8822fc7cb348b" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 32947 } }
I20260620 14:47:27.150213 2652 raft_consensus.cc:374] T 70245e00c5ed4751a2f7d04dc7dac140 P 5aa58728f3b74083b017cdcd573bc626 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 14:47:27.150309 2652 raft_consensus.cc:729] T 70245e00c5ed4751a2f7d04dc7dac140 P 5aa58728f3b74083b017cdcd573bc626 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5aa58728f3b74083b017cdcd573bc626, State: Initialized, Role: FOLLOWER
I20260620 14:47:27.150565 2652 consensus_queue.cc:260] T 70245e00c5ed4751a2f7d04dc7dac140 P 5aa58728f3b74083b017cdcd573bc626 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5aa58728f3b74083b017cdcd573bc626" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 35927 } } peers { permanent_uuid: "e5717b124bc7489e8817304e3108a0eb" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 37155 } } peers { permanent_uuid: "7f1b44e3bb694105afe8822fc7cb348b" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 32947 } }
I20260620 14:47:27.151813 2652 ts_tablet_manager.cc:1428] T 70245e00c5ed4751a2f7d04dc7dac140 P 5aa58728f3b74083b017cdcd573bc626: Time spent starting tablet: real 0.003s user 0.002s sys 0.001s
I20260620 14:47:27.156685 2698 ts_tablet_manager.cc:1918] T f2f5ceda90ee40a9b777efa5d4309c1a P e5717b124bc7489e8817304e3108a0eb: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260620 14:47:27.156867 2698 log.cc:1196] T f2f5ceda90ee40a9b777efa5d4309c1a P e5717b124bc7489e8817304e3108a0eb: Deleting WAL directory at /tmp/dist-test-taskUci3DU/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781966829091300-388-0/minicluster-data/ts-1/wal/wals/f2f5ceda90ee40a9b777efa5d4309c1a
I20260620 14:47:27.157197 2649 tablet_bootstrap.cc:492] T 70245e00c5ed4751a2f7d04dc7dac140 P e5717b124bc7489e8817304e3108a0eb: No bootstrap required, opened a new log
I20260620 14:47:27.157377 2649 ts_tablet_manager.cc:1397] T 70245e00c5ed4751a2f7d04dc7dac140 P e5717b124bc7489e8817304e3108a0eb: Time spent bootstrapping tablet: real 0.024s user 0.002s sys 0.008s
I20260620 14:47:27.157434 2698 ts_tablet_manager.cc:1939] T f2f5ceda90ee40a9b777efa5d4309c1a P e5717b124bc7489e8817304e3108a0eb: Deleting consensus metadata
I20260620 14:47:27.158218 2649 raft_consensus.cc:348] T 70245e00c5ed4751a2f7d04dc7dac140 P e5717b124bc7489e8817304e3108a0eb [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5aa58728f3b74083b017cdcd573bc626" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 35927 } } peers { permanent_uuid: "e5717b124bc7489e8817304e3108a0eb" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 37155 } } peers { permanent_uuid: "7f1b44e3bb694105afe8822fc7cb348b" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 32947 } }
I20260620 14:47:27.158483 2649 raft_consensus.cc:374] T 70245e00c5ed4751a2f7d04dc7dac140 P e5717b124bc7489e8817304e3108a0eb [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 14:47:27.158565 2649 raft_consensus.cc:729] T 70245e00c5ed4751a2f7d04dc7dac140 P e5717b124bc7489e8817304e3108a0eb [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e5717b124bc7489e8817304e3108a0eb, State: Initialized, Role: FOLLOWER
I20260620 14:47:27.159165 2129 catalog_manager.cc:4841] TS e5717b124bc7489e8817304e3108a0eb (127.0.97.2:37155): tablet f2f5ceda90ee40a9b777efa5d4309c1a (table test-table [id=58708c43b2174db18b083fae06190952]) successfully deleted
I20260620 14:47:27.158859 2649 consensus_queue.cc:260] T 70245e00c5ed4751a2f7d04dc7dac140 P e5717b124bc7489e8817304e3108a0eb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5aa58728f3b74083b017cdcd573bc626" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 35927 } } peers { permanent_uuid: "e5717b124bc7489e8817304e3108a0eb" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 37155 } } peers { permanent_uuid: "7f1b44e3bb694105afe8822fc7cb348b" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 32947 } }
I20260620 14:47:27.159857 2649 ts_tablet_manager.cc:1428] T 70245e00c5ed4751a2f7d04dc7dac140 P e5717b124bc7489e8817304e3108a0eb: Time spent starting tablet: real 0.002s user 0.001s sys 0.001s
I20260620 14:47:27.164077 2699 ts_tablet_manager.cc:1918] T f2f5ceda90ee40a9b777efa5d4309c1a P 5aa58728f3b74083b017cdcd573bc626: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260620 14:47:27.164268 2699 log.cc:1196] T f2f5ceda90ee40a9b777efa5d4309c1a P 5aa58728f3b74083b017cdcd573bc626: Deleting WAL directory at /tmp/dist-test-taskUci3DU/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781966829091300-388-0/minicluster-data/ts-0/wal/wals/f2f5ceda90ee40a9b777efa5d4309c1a
I20260620 14:47:27.164799 2699 ts_tablet_manager.cc:1939] T f2f5ceda90ee40a9b777efa5d4309c1a P 5aa58728f3b74083b017cdcd573bc626: Deleting consensus metadata
I20260620 14:47:27.166688 2130 catalog_manager.cc:4841] TS 5aa58728f3b74083b017cdcd573bc626 (127.0.97.1:35927): tablet f2f5ceda90ee40a9b777efa5d4309c1a (table test-table [id=58708c43b2174db18b083fae06190952]) successfully deleted
I20260620 14:47:27.256567 2656 raft_consensus.cc:482] T 70245e00c5ed4751a2f7d04dc7dac140 P 7f1b44e3bb694105afe8822fc7cb348b [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260620 14:47:27.256815 2656 raft_consensus.cc:504] T 70245e00c5ed4751a2f7d04dc7dac140 P 7f1b44e3bb694105afe8822fc7cb348b [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5aa58728f3b74083b017cdcd573bc626" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 35927 } } peers { permanent_uuid: "e5717b124bc7489e8817304e3108a0eb" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 37155 } } peers { permanent_uuid: "7f1b44e3bb694105afe8822fc7cb348b" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 32947 } }
I20260620 14:47:27.257611 2656 leader_election.cc:288] T 70245e00c5ed4751a2f7d04dc7dac140 P 7f1b44e3bb694105afe8822fc7cb348b [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 5aa58728f3b74083b017cdcd573bc626 (127.0.97.1:35927), e5717b124bc7489e8817304e3108a0eb (127.0.97.2:37155)
I20260620 14:47:27.258270 2302 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "70245e00c5ed4751a2f7d04dc7dac140" candidate_uuid: "7f1b44e3bb694105afe8822fc7cb348b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5aa58728f3b74083b017cdcd573bc626" is_pre_election: true
I20260620 14:47:27.258354 2441 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "70245e00c5ed4751a2f7d04dc7dac140" candidate_uuid: "7f1b44e3bb694105afe8822fc7cb348b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e5717b124bc7489e8817304e3108a0eb" is_pre_election: true
I20260620 14:47:27.258548 2302 raft_consensus.cc:2454] T 70245e00c5ed4751a2f7d04dc7dac140 P 5aa58728f3b74083b017cdcd573bc626 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7f1b44e3bb694105afe8822fc7cb348b in term 0.
I20260620 14:47:27.258601 2441 raft_consensus.cc:2454] T 70245e00c5ed4751a2f7d04dc7dac140 P e5717b124bc7489e8817304e3108a0eb [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7f1b44e3bb694105afe8822fc7cb348b in term 0.
I20260620 14:47:27.259097 2512 leader_election.cc:302] T 70245e00c5ed4751a2f7d04dc7dac140 P 7f1b44e3bb694105afe8822fc7cb348b [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 3 yes votes; 0 no votes. yes voters: 5aa58728f3b74083b017cdcd573bc626, 7f1b44e3bb694105afe8822fc7cb348b, e5717b124bc7489e8817304e3108a0eb; no voters:
I20260620 14:47:27.259517 2656 raft_consensus.cc:2781] T 70245e00c5ed4751a2f7d04dc7dac140 P 7f1b44e3bb694105afe8822fc7cb348b [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260620 14:47:27.259668 2656 raft_consensus.cc:482] T 70245e00c5ed4751a2f7d04dc7dac140 P 7f1b44e3bb694105afe8822fc7cb348b [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260620 14:47:27.259764 2656 raft_consensus.cc:3037] T 70245e00c5ed4751a2f7d04dc7dac140 P 7f1b44e3bb694105afe8822fc7cb348b [term 0 FOLLOWER]: Advancing to term 1
I20260620 14:47:27.263078 2656 raft_consensus.cc:504] T 70245e00c5ed4751a2f7d04dc7dac140 P 7f1b44e3bb694105afe8822fc7cb348b [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5aa58728f3b74083b017cdcd573bc626" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 35927 } } peers { permanent_uuid: "e5717b124bc7489e8817304e3108a0eb" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 37155 } } peers { permanent_uuid: "7f1b44e3bb694105afe8822fc7cb348b" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 32947 } }
I20260620 14:47:27.264339 2441 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "70245e00c5ed4751a2f7d04dc7dac140" candidate_uuid: "7f1b44e3bb694105afe8822fc7cb348b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e5717b124bc7489e8817304e3108a0eb"
I20260620 14:47:27.264360 2302 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "70245e00c5ed4751a2f7d04dc7dac140" candidate_uuid: "7f1b44e3bb694105afe8822fc7cb348b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5aa58728f3b74083b017cdcd573bc626"
I20260620 14:47:27.264570 2441 raft_consensus.cc:3037] T 70245e00c5ed4751a2f7d04dc7dac140 P e5717b124bc7489e8817304e3108a0eb [term 0 FOLLOWER]: Advancing to term 1
I20260620 14:47:27.264578 2302 raft_consensus.cc:3037] T 70245e00c5ed4751a2f7d04dc7dac140 P 5aa58728f3b74083b017cdcd573bc626 [term 0 FOLLOWER]: Advancing to term 1
I20260620 14:47:27.267711 2441 raft_consensus.cc:2454] T 70245e00c5ed4751a2f7d04dc7dac140 P e5717b124bc7489e8817304e3108a0eb [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7f1b44e3bb694105afe8822fc7cb348b in term 1.
I20260620 14:47:27.267952 2302 raft_consensus.cc:2454] T 70245e00c5ed4751a2f7d04dc7dac140 P 5aa58728f3b74083b017cdcd573bc626 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7f1b44e3bb694105afe8822fc7cb348b in term 1.
I20260620 14:47:27.268324 2511 leader_election.cc:302] T 70245e00c5ed4751a2f7d04dc7dac140 P 7f1b44e3bb694105afe8822fc7cb348b [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 7f1b44e3bb694105afe8822fc7cb348b, e5717b124bc7489e8817304e3108a0eb; no voters:
I20260620 14:47:27.268702 2672 raft_consensus.cc:2781] T 70245e00c5ed4751a2f7d04dc7dac140 P 7f1b44e3bb694105afe8822fc7cb348b [term 1 FOLLOWER]: Leader election won for term 1
I20260620 14:47:27.268877 2672 raft_consensus.cc:686] T 70245e00c5ed4751a2f7d04dc7dac140 P 7f1b44e3bb694105afe8822fc7cb348b [term 1 LEADER]: Becoming Leader. State: Replica: 7f1b44e3bb694105afe8822fc7cb348b, State: Running, Role: LEADER
I20260620 14:47:27.269225 2672 consensus_queue.cc:237] T 70245e00c5ed4751a2f7d04dc7dac140 P 7f1b44e3bb694105afe8822fc7cb348b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5aa58728f3b74083b017cdcd573bc626" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 35927 } } peers { permanent_uuid: "e5717b124bc7489e8817304e3108a0eb" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 37155 } } peers { permanent_uuid: "7f1b44e3bb694105afe8822fc7cb348b" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 32947 } }
I20260620 14:47:27.271513 2656 leader_election.cc:288] T 70245e00c5ed4751a2f7d04dc7dac140 P 7f1b44e3bb694105afe8822fc7cb348b [CANDIDATE]: Term 1 election: Requested vote from peers 5aa58728f3b74083b017cdcd573bc626 (127.0.97.1:35927), e5717b124bc7489e8817304e3108a0eb (127.0.97.2:37155)
I20260620 14:47:27.278016 2143 catalog_manager.cc:5495] T 70245e00c5ed4751a2f7d04dc7dac140 P 7f1b44e3bb694105afe8822fc7cb348b reported cstate change: term changed from 0 to 1, leader changed from <none> to 7f1b44e3bb694105afe8822fc7cb348b (127.0.97.3). New cstate: current_term: 1 leader_uuid: "7f1b44e3bb694105afe8822fc7cb348b" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5aa58728f3b74083b017cdcd573bc626" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 35927 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "e5717b124bc7489e8817304e3108a0eb" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 37155 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "7f1b44e3bb694105afe8822fc7cb348b" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 32947 } health_report { overall_health: HEALTHY } } }
I20260620 14:47:27.317335 2646 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:47:27.308264 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46344 (local address 127.0.97.3:32947)
0620 14:47:27.308436 (+ 172us) server_negotiation.cc:206] Beginning negotiation
0620 14:47:27.308443 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0620 14:47:27.308463 (+ 20us) server_negotiation.cc:407] Connection header received
0620 14:47:27.308527 (+ 64us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:47:27.308532 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:47:27.308621 (+ 89us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:47:27.308723 (+ 102us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 14:47:27.309191 (+ 468us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:47:27.310435 (+ 1244us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:47:27.312938 (+ 2503us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:47:27.313647 (+ 709us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:47:27.313771 (+ 124us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:47:27.314378 (+ 607us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 14:47:27.314552 (+ 174us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0620 14:47:27.314652 (+ 100us) server_negotiation.cc:1035] Waiting for connection context
0620 14:47:27.316902 (+ 2250us) server_negotiation.cc:299] Negotiation successful
0620 14:47:27.316983 (+ 81us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":46}
I20260620 14:47:27.334457 2646 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:47:27.327730 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.3:33953 (local address 127.0.97.3:32947)
0620 14:47:27.327923 (+ 193us) server_negotiation.cc:206] Beginning negotiation
0620 14:47:27.327929 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 14:47:27.328472 (+ 543us) server_negotiation.cc:407] Connection header received
0620 14:47:27.328531 (+ 59us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:47:27.328536 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:47:27.328610 (+ 74us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:47:27.328692 (+ 82us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 14:47:27.329168 (+ 476us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:47:27.330333 (+ 1165us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:47:27.332371 (+ 2038us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:47:27.333050 (+ 679us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:47:27.333178 (+ 128us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:47:27.333606 (+ 428us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 14:47:27.333733 (+ 127us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0620 14:47:27.333835 (+ 102us) server_negotiation.cc:1035] Waiting for connection context
0620 14:47:27.333981 (+ 146us) server_negotiation.cc:299] Negotiation successful
0620 14:47:27.334090 (+ 109us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":21,"server-negotiator.queue_time_us":43}
I20260620 14:47:27.334489 2694 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:47:27.327772 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.3:32947 (local address 127.0.97.3:33953)
0620 14:47:27.328159 (+ 387us) negotiation.cc:107] Waiting for socket to connect
0620 14:47:27.328182 (+ 23us) client_negotiation.cc:174] Beginning negotiation
0620 14:47:27.328308 (+ 126us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 14:47:27.328751 (+ 443us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 14:47:27.328756 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 14:47:27.328769 (+ 13us) client_negotiation.cc:189] Negotiated authn=TOKEN
0620 14:47:27.328983 (+ 214us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:47:27.328997 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:47:27.330495 (+ 1498us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:47:27.330501 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:47:27.332217 (+ 1716us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:47:27.332234 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:47:27.333203 (+ 969us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:47:27.333209 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:47:27.333386 (+ 177us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:47:27.333435 (+ 49us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0620 14:47:27.333849 (+ 414us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0620 14:47:27.333861 (+ 12us) client_negotiation.cc:769] Sending connection context
0620 14:47:27.333952 (+ 91us) client_negotiation.cc:240] Negotiation successful
0620 14:47:27.334118 (+ 166us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":239,"mutex_wait_us":1}
I20260620 14:47:27.346961 2441 raft_consensus.cc:1261] T 70245e00c5ed4751a2f7d04dc7dac140 P e5717b124bc7489e8817304e3108a0eb [term 1 FOLLOWER]: Refusing update from remote peer 7f1b44e3bb694105afe8822fc7cb348b: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 14:47:27.347553 2672 consensus_queue.cc:1035] T 70245e00c5ed4751a2f7d04dc7dac140 P 7f1b44e3bb694105afe8822fc7cb348b [LEADER]: Connected to new peer: Peer: permanent_uuid: "e5717b124bc7489e8817304e3108a0eb" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 37155 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 14:47:27.348001 2302 raft_consensus.cc:1261] T 70245e00c5ed4751a2f7d04dc7dac140 P 5aa58728f3b74083b017cdcd573bc626 [term 1 FOLLOWER]: Refusing update from remote peer 7f1b44e3bb694105afe8822fc7cb348b: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 14:47:27.349910 2672 consensus_queue.cc:1035] T 70245e00c5ed4751a2f7d04dc7dac140 P 7f1b44e3bb694105afe8822fc7cb348b [LEADER]: Connected to new peer: Peer: permanent_uuid: "5aa58728f3b74083b017cdcd573bc626" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 35927 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 14:47:27.356557 2678 mvcc.cc:205] Tried to move back new op lower bound from 7298936206728142848 to 7298936206417526784. Current Snapshot: MvccSnapshot[applied={T|T < 7298936206728142848}]
W20260620 14:47:27.381482 2677 tablet_replica.cc:1298] Aborted: operation has been aborted: cancelling pending write operations
I20260620 14:47:29.003391 2143 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:40400:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 14:47:29.003659 2143 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:40400:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 14:47:29.007496 2143 catalog_manager.cc:5782] T 00000000000000000000000000000000 P dbf70d79ae0b41f28a6637ed9ab0ecc4: Sending DeleteTablet for 3 replicas of tablet 70245e00c5ed4751a2f7d04dc7dac140
I20260620 14:47:29.008359 2281 tablet_service.cc:1514] Processing DeleteTablet for tablet 70245e00c5ed4751a2f7d04dc7dac140 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 14:47:29 UTC) from {username='kudu', principal='kudu/127.0.97.62@KRBTEST.COM'} at 127.0.0.1:48790
I20260620 14:47:29.008399 2560 tablet_service.cc:1514] Processing DeleteTablet for tablet 70245e00c5ed4751a2f7d04dc7dac140 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 14:47:29 UTC) from {username='kudu', principal='kudu/127.0.97.62@KRBTEST.COM'} at 127.0.0.1:46328
I20260620 14:47:29.008397 2420 tablet_service.cc:1514] Processing DeleteTablet for tablet 70245e00c5ed4751a2f7d04dc7dac140 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 14:47:29 UTC) from {username='kudu', principal='kudu/127.0.97.62@KRBTEST.COM'} at 127.0.0.1:41730
I20260620 14:47:29.009485 2728 tablet_replica.cc:330] T 70245e00c5ed4751a2f7d04dc7dac140 P e5717b124bc7489e8817304e3108a0eb: stopping tablet replica
I20260620 14:47:29.009824 2728 raft_consensus.cc:2229] T 70245e00c5ed4751a2f7d04dc7dac140 P e5717b124bc7489e8817304e3108a0eb [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 14:47:29.011181 2728 raft_consensus.cc:2258] T 70245e00c5ed4751a2f7d04dc7dac140 P e5717b124bc7489e8817304e3108a0eb [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 14:47:29.012856 2728 ts_tablet_manager.cc:1905] T 70245e00c5ed4751a2f7d04dc7dac140 P e5717b124bc7489e8817304e3108a0eb: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 14:47:29.021896 2729 tablet_replica.cc:330] T 70245e00c5ed4751a2f7d04dc7dac140 P 7f1b44e3bb694105afe8822fc7cb348b: stopping tablet replica
I20260620 14:47:29.022119 2727 tablet_replica.cc:330] T 70245e00c5ed4751a2f7d04dc7dac140 P 5aa58728f3b74083b017cdcd573bc626: stopping tablet replica
I20260620 14:47:29.022145 2728 ts_tablet_manager.cc:1918] T 70245e00c5ed4751a2f7d04dc7dac140 P e5717b124bc7489e8817304e3108a0eb: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.8
I20260620 14:47:29.022295 2728 log.cc:1196] T 70245e00c5ed4751a2f7d04dc7dac140 P e5717b124bc7489e8817304e3108a0eb: Deleting WAL directory at /tmp/dist-test-taskUci3DU/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781966829091300-388-0/minicluster-data/ts-1/wal/wals/70245e00c5ed4751a2f7d04dc7dac140
I20260620 14:47:29.022256 2729 raft_consensus.cc:2229] T 70245e00c5ed4751a2f7d04dc7dac140 P 7f1b44e3bb694105afe8822fc7cb348b [term 1 LEADER]: Raft consensus shutting down.
I20260620 14:47:29.022477 2727 raft_consensus.cc:2229] T 70245e00c5ed4751a2f7d04dc7dac140 P 5aa58728f3b74083b017cdcd573bc626 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 14:47:29.022778 2727 raft_consensus.cc:2258] T 70245e00c5ed4751a2f7d04dc7dac140 P 5aa58728f3b74083b017cdcd573bc626 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 14:47:29.022791 2728 ts_tablet_manager.cc:1939] T 70245e00c5ed4751a2f7d04dc7dac140 P e5717b124bc7489e8817304e3108a0eb: Deleting consensus metadata
I20260620 14:47:29.022874 2729 raft_consensus.cc:2258] T 70245e00c5ed4751a2f7d04dc7dac140 P 7f1b44e3bb694105afe8822fc7cb348b [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 14:47:29.023907 2727 ts_tablet_manager.cc:1905] T 70245e00c5ed4751a2f7d04dc7dac140 P 5aa58728f3b74083b017cdcd573bc626: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 14:47:29.023926 2729 ts_tablet_manager.cc:1905] T 70245e00c5ed4751a2f7d04dc7dac140 P 7f1b44e3bb694105afe8822fc7cb348b: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 14:47:29.024287 2129 catalog_manager.cc:4841] TS e5717b124bc7489e8817304e3108a0eb (127.0.97.2:37155): tablet 70245e00c5ed4751a2f7d04dc7dac140 (table test-table [id=243167de760e48f382fe3b25335c6a85]) successfully deleted
I20260620 14:47:29.028760 2727 ts_tablet_manager.cc:1918] T 70245e00c5ed4751a2f7d04dc7dac140 P 5aa58728f3b74083b017cdcd573bc626: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.8
I20260620 14:47:29.028913 2727 log.cc:1196] T 70245e00c5ed4751a2f7d04dc7dac140 P 5aa58728f3b74083b017cdcd573bc626: Deleting WAL directory at /tmp/dist-test-taskUci3DU/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781966829091300-388-0/minicluster-data/ts-0/wal/wals/70245e00c5ed4751a2f7d04dc7dac140
I20260620 14:47:29.029363 2727 ts_tablet_manager.cc:1939] T 70245e00c5ed4751a2f7d04dc7dac140 P 5aa58728f3b74083b017cdcd573bc626: Deleting consensus metadata
I20260620 14:47:29.030373 2130 catalog_manager.cc:4841] TS 5aa58728f3b74083b017cdcd573bc626 (127.0.97.1:35927): tablet 70245e00c5ed4751a2f7d04dc7dac140 (table test-table [id=243167de760e48f382fe3b25335c6a85]) successfully deleted
I20260620 14:47:29.031947 2729 ts_tablet_manager.cc:1918] T 70245e00c5ed4751a2f7d04dc7dac140 P 7f1b44e3bb694105afe8822fc7cb348b: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.8
I20260620 14:47:29.032085 2729 log.cc:1196] T 70245e00c5ed4751a2f7d04dc7dac140 P 7f1b44e3bb694105afe8822fc7cb348b: Deleting WAL directory at /tmp/dist-test-taskUci3DU/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781966829091300-388-0/minicluster-data/ts-2/wal/wals/70245e00c5ed4751a2f7d04dc7dac140
I20260620 14:47:29.032620 2729 ts_tablet_manager.cc:1939] T 70245e00c5ed4751a2f7d04dc7dac140 P 7f1b44e3bb694105afe8822fc7cb348b: Deleting consensus metadata
I20260620 14:47:29.033823 2130 catalog_manager.cc:4841] TS 7f1b44e3bb694105afe8822fc7cb348b (127.0.97.3:32947): tablet 70245e00c5ed4751a2f7d04dc7dac140 (table test-table [id=243167de760e48f382fe3b25335c6a85]) successfully deleted
Jun 20 14:47:29 dist-test-slave-q9j6 krb5kdc[2100](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781966849, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-user@KRBTEST.COM:
Jun 20 14:47:29 dist-test-slave-q9j6 krb5kdc[2100](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781966849, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260620 14:47:29.067510 2737 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:47:29.053446 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40420 (local address 127.0.97.62:37975)
0620 14:47:29.054032 (+ 586us) server_negotiation.cc:206] Beginning negotiation
0620 14:47:29.054039 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0620 14:47:29.054064 (+ 25us) server_negotiation.cc:407] Connection header received
0620 14:47:29.054280 (+ 216us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:47:29.054284 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:47:29.054345 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:47:29.054459 (+ 114us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:47:29.055798 (+ 1339us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:47:29.056540 (+ 742us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:47:29.057789 (+ 1249us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:47:29.058208 (+ 419us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:47:29.058352 (+ 144us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:47:29.062424 (+ 4072us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 14:47:29.062429 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 14:47:29.062433 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 14:47:29.062472 (+ 39us) server_negotiation.cc:922] calling sasl_server_start()
0620 14:47:29.065185 (+ 2713us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:47:29.065867 (+ 682us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:47:29.065870 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:47:29.065873 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:47:29.065996 (+ 123us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:47:29.066292 (+ 296us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:47:29.066295 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:47:29.066297 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:47:29.066543 (+ 246us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 14:47:29.066709 (+ 166us) server_negotiation.cc:1035] Waiting for connection context
0620 14:47:29.066970 (+ 261us) server_negotiation.cc:299] Negotiation successful
0620 14:47:29.067129 (+ 159us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":454,"thread_start_us":320,"threads_started":1}
I20260620 14:47:29.070829 2143 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:40420:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260620 14:47:29.071413 2143 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260620 14:47:29.077495 2281 tablet_service.cc:1467] Processing CreateTablet for tablet 6121447adcd045d7b4d20c85cd37d891 (DEFAULT_TABLE table=test-table [id=72c70197be594173b771df013c32267f]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 14:47:29.077656 2420 tablet_service.cc:1467] Processing CreateTablet for tablet 6121447adcd045d7b4d20c85cd37d891 (DEFAULT_TABLE table=test-table [id=72c70197be594173b771df013c32267f]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 14:47:29.077870 2560 tablet_service.cc:1467] Processing CreateTablet for tablet 6121447adcd045d7b4d20c85cd37d891 (DEFAULT_TABLE table=test-table [id=72c70197be594173b771df013c32267f]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 14:47:29.077992 2281 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6121447adcd045d7b4d20c85cd37d891. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 14:47:29.078032 2420 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6121447adcd045d7b4d20c85cd37d891. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 14:47:29.078287 2560 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6121447adcd045d7b4d20c85cd37d891. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 14:47:29.084668 2738 tablet_bootstrap.cc:492] T 6121447adcd045d7b4d20c85cd37d891 P 7f1b44e3bb694105afe8822fc7cb348b: Bootstrap starting.
I20260620 14:47:29.085431 2739 tablet_bootstrap.cc:492] T 6121447adcd045d7b4d20c85cd37d891 P e5717b124bc7489e8817304e3108a0eb: Bootstrap starting.
I20260620 14:47:29.086258 2740 tablet_bootstrap.cc:492] T 6121447adcd045d7b4d20c85cd37d891 P 5aa58728f3b74083b017cdcd573bc626: Bootstrap starting.
I20260620 14:47:29.087195 2739 tablet_bootstrap.cc:654] T 6121447adcd045d7b4d20c85cd37d891 P e5717b124bc7489e8817304e3108a0eb: Neither blocks nor log segments found. Creating new log.
I20260620 14:47:29.087744 2738 tablet_bootstrap.cc:654] T 6121447adcd045d7b4d20c85cd37d891 P 7f1b44e3bb694105afe8822fc7cb348b: Neither blocks nor log segments found. Creating new log.
I20260620 14:47:29.088667 2740 tablet_bootstrap.cc:654] T 6121447adcd045d7b4d20c85cd37d891 P 5aa58728f3b74083b017cdcd573bc626: Neither blocks nor log segments found. Creating new log.
I20260620 14:47:29.089823 2739 tablet_bootstrap.cc:492] T 6121447adcd045d7b4d20c85cd37d891 P e5717b124bc7489e8817304e3108a0eb: No bootstrap required, opened a new log
I20260620 14:47:29.089989 2739 ts_tablet_manager.cc:1397] T 6121447adcd045d7b4d20c85cd37d891 P e5717b124bc7489e8817304e3108a0eb: Time spent bootstrapping tablet: real 0.005s user 0.000s sys 0.003s
I20260620 14:47:29.090126 2738 tablet_bootstrap.cc:492] T 6121447adcd045d7b4d20c85cd37d891 P 7f1b44e3bb694105afe8822fc7cb348b: No bootstrap required, opened a new log
I20260620 14:47:29.090276 2738 ts_tablet_manager.cc:1397] T 6121447adcd045d7b4d20c85cd37d891 P 7f1b44e3bb694105afe8822fc7cb348b: Time spent bootstrapping tablet: real 0.006s user 0.003s sys 0.000s
I20260620 14:47:29.090741 2740 tablet_bootstrap.cc:492] T 6121447adcd045d7b4d20c85cd37d891 P 5aa58728f3b74083b017cdcd573bc626: No bootstrap required, opened a new log
I20260620 14:47:29.090884 2740 ts_tablet_manager.cc:1397] T 6121447adcd045d7b4d20c85cd37d891 P 5aa58728f3b74083b017cdcd573bc626: Time spent bootstrapping tablet: real 0.005s user 0.003s sys 0.000s
I20260620 14:47:29.090726 2739 raft_consensus.cc:348] T 6121447adcd045d7b4d20c85cd37d891 P e5717b124bc7489e8817304e3108a0eb [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5aa58728f3b74083b017cdcd573bc626" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 35927 } } peers { permanent_uuid: "e5717b124bc7489e8817304e3108a0eb" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 37155 } } peers { permanent_uuid: "7f1b44e3bb694105afe8822fc7cb348b" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 32947 } }
I20260620 14:47:29.090982 2739 raft_consensus.cc:374] T 6121447adcd045d7b4d20c85cd37d891 P e5717b124bc7489e8817304e3108a0eb [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 14:47:29.091078 2739 raft_consensus.cc:729] T 6121447adcd045d7b4d20c85cd37d891 P e5717b124bc7489e8817304e3108a0eb [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e5717b124bc7489e8817304e3108a0eb, State: Initialized, Role: FOLLOWER
I20260620 14:47:29.091115 2738 raft_consensus.cc:348] T 6121447adcd045d7b4d20c85cd37d891 P 7f1b44e3bb694105afe8822fc7cb348b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5aa58728f3b74083b017cdcd573bc626" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 35927 } } peers { permanent_uuid: "e5717b124bc7489e8817304e3108a0eb" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 37155 } } peers { permanent_uuid: "7f1b44e3bb694105afe8822fc7cb348b" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 32947 } }
I20260620 14:47:29.091343 2738 raft_consensus.cc:374] T 6121447adcd045d7b4d20c85cd37d891 P 7f1b44e3bb694105afe8822fc7cb348b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 14:47:29.091425 2738 raft_consensus.cc:729] T 6121447adcd045d7b4d20c85cd37d891 P 7f1b44e3bb694105afe8822fc7cb348b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7f1b44e3bb694105afe8822fc7cb348b, State: Initialized, Role: FOLLOWER
I20260620 14:47:29.091327 2739 consensus_queue.cc:260] T 6121447adcd045d7b4d20c85cd37d891 P e5717b124bc7489e8817304e3108a0eb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5aa58728f3b74083b017cdcd573bc626" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 35927 } } peers { permanent_uuid: "e5717b124bc7489e8817304e3108a0eb" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 37155 } } peers { permanent_uuid: "7f1b44e3bb694105afe8822fc7cb348b" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 32947 } }
I20260620 14:47:29.091674 2738 consensus_queue.cc:260] T 6121447adcd045d7b4d20c85cd37d891 P 7f1b44e3bb694105afe8822fc7cb348b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5aa58728f3b74083b017cdcd573bc626" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 35927 } } peers { permanent_uuid: "e5717b124bc7489e8817304e3108a0eb" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 37155 } } peers { permanent_uuid: "7f1b44e3bb694105afe8822fc7cb348b" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 32947 } }
I20260620 14:47:29.092283 2738 ts_tablet_manager.cc:1428] T 6121447adcd045d7b4d20c85cd37d891 P 7f1b44e3bb694105afe8822fc7cb348b: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20260620 14:47:29.092769 2740 raft_consensus.cc:348] T 6121447adcd045d7b4d20c85cd37d891 P 5aa58728f3b74083b017cdcd573bc626 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5aa58728f3b74083b017cdcd573bc626" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 35927 } } peers { permanent_uuid: "e5717b124bc7489e8817304e3108a0eb" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 37155 } } peers { permanent_uuid: "7f1b44e3bb694105afe8822fc7cb348b" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 32947 } }
I20260620 14:47:29.092994 2740 raft_consensus.cc:374] T 6121447adcd045d7b4d20c85cd37d891 P 5aa58728f3b74083b017cdcd573bc626 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 14:47:29.093091 2740 raft_consensus.cc:729] T 6121447adcd045d7b4d20c85cd37d891 P 5aa58728f3b74083b017cdcd573bc626 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5aa58728f3b74083b017cdcd573bc626, State: Initialized, Role: FOLLOWER
I20260620 14:47:29.093346 2740 consensus_queue.cc:260] T 6121447adcd045d7b4d20c85cd37d891 P 5aa58728f3b74083b017cdcd573bc626 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5aa58728f3b74083b017cdcd573bc626" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 35927 } } peers { permanent_uuid: "e5717b124bc7489e8817304e3108a0eb" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 37155 } } peers { permanent_uuid: "7f1b44e3bb694105afe8822fc7cb348b" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 32947 } }
I20260620 14:47:29.094645 2740 ts_tablet_manager.cc:1428] T 6121447adcd045d7b4d20c85cd37d891 P 5aa58728f3b74083b017cdcd573bc626: Time spent starting tablet: real 0.004s user 0.002s sys 0.000s
I20260620 14:47:29.096243 2739 ts_tablet_manager.cc:1428] T 6121447adcd045d7b4d20c85cd37d891 P e5717b124bc7489e8817304e3108a0eb: Time spent starting tablet: real 0.006s user 0.000s sys 0.006s
I20260620 14:47:29.108790 2744 raft_consensus.cc:482] T 6121447adcd045d7b4d20c85cd37d891 P e5717b124bc7489e8817304e3108a0eb [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260620 14:47:29.109028 2744 raft_consensus.cc:504] T 6121447adcd045d7b4d20c85cd37d891 P e5717b124bc7489e8817304e3108a0eb [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5aa58728f3b74083b017cdcd573bc626" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 35927 } } peers { permanent_uuid: "e5717b124bc7489e8817304e3108a0eb" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 37155 } } peers { permanent_uuid: "7f1b44e3bb694105afe8822fc7cb348b" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 32947 } }
I20260620 14:47:29.109789 2744 leader_election.cc:288] T 6121447adcd045d7b4d20c85cd37d891 P e5717b124bc7489e8817304e3108a0eb [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 5aa58728f3b74083b017cdcd573bc626 (127.0.97.1:35927), 7f1b44e3bb694105afe8822fc7cb348b (127.0.97.3:32947)
I20260620 14:47:29.110348 2301 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6121447adcd045d7b4d20c85cd37d891" candidate_uuid: "e5717b124bc7489e8817304e3108a0eb" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5aa58728f3b74083b017cdcd573bc626" is_pre_election: true
I20260620 14:47:29.110399 2580 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6121447adcd045d7b4d20c85cd37d891" candidate_uuid: "e5717b124bc7489e8817304e3108a0eb" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7f1b44e3bb694105afe8822fc7cb348b" is_pre_election: true
I20260620 14:47:29.110591 2301 raft_consensus.cc:2454] T 6121447adcd045d7b4d20c85cd37d891 P 5aa58728f3b74083b017cdcd573bc626 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e5717b124bc7489e8817304e3108a0eb in term 0.
I20260620 14:47:29.110668 2580 raft_consensus.cc:2454] T 6121447adcd045d7b4d20c85cd37d891 P 7f1b44e3bb694105afe8822fc7cb348b [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e5717b124bc7489e8817304e3108a0eb in term 0.
I20260620 14:47:29.111064 2373 leader_election.cc:302] T 6121447adcd045d7b4d20c85cd37d891 P e5717b124bc7489e8817304e3108a0eb [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5aa58728f3b74083b017cdcd573bc626, e5717b124bc7489e8817304e3108a0eb; no voters:
I20260620 14:47:29.111321 2744 raft_consensus.cc:2781] T 6121447adcd045d7b4d20c85cd37d891 P e5717b124bc7489e8817304e3108a0eb [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260620 14:47:29.111447 2744 raft_consensus.cc:482] T 6121447adcd045d7b4d20c85cd37d891 P e5717b124bc7489e8817304e3108a0eb [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260620 14:47:29.111544 2744 raft_consensus.cc:3037] T 6121447adcd045d7b4d20c85cd37d891 P e5717b124bc7489e8817304e3108a0eb [term 0 FOLLOWER]: Advancing to term 1
I20260620 14:47:29.113701 2744 raft_consensus.cc:504] T 6121447adcd045d7b4d20c85cd37d891 P e5717b124bc7489e8817304e3108a0eb [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5aa58728f3b74083b017cdcd573bc626" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 35927 } } peers { permanent_uuid: "e5717b124bc7489e8817304e3108a0eb" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 37155 } } peers { permanent_uuid: "7f1b44e3bb694105afe8822fc7cb348b" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 32947 } }
I20260620 14:47:29.114319 2744 leader_election.cc:288] T 6121447adcd045d7b4d20c85cd37d891 P e5717b124bc7489e8817304e3108a0eb [CANDIDATE]: Term 1 election: Requested vote from peers 5aa58728f3b74083b017cdcd573bc626 (127.0.97.1:35927), 7f1b44e3bb694105afe8822fc7cb348b (127.0.97.3:32947)
I20260620 14:47:29.114821 2301 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6121447adcd045d7b4d20c85cd37d891" candidate_uuid: "e5717b124bc7489e8817304e3108a0eb" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5aa58728f3b74083b017cdcd573bc626"
I20260620 14:47:29.114842 2580 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6121447adcd045d7b4d20c85cd37d891" candidate_uuid: "e5717b124bc7489e8817304e3108a0eb" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7f1b44e3bb694105afe8822fc7cb348b"
I20260620 14:47:29.115044 2580 raft_consensus.cc:3037] T 6121447adcd045d7b4d20c85cd37d891 P 7f1b44e3bb694105afe8822fc7cb348b [term 0 FOLLOWER]: Advancing to term 1
I20260620 14:47:29.115044 2301 raft_consensus.cc:3037] T 6121447adcd045d7b4d20c85cd37d891 P 5aa58728f3b74083b017cdcd573bc626 [term 0 FOLLOWER]: Advancing to term 1
I20260620 14:47:29.118047 2301 raft_consensus.cc:2454] T 6121447adcd045d7b4d20c85cd37d891 P 5aa58728f3b74083b017cdcd573bc626 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e5717b124bc7489e8817304e3108a0eb in term 1.
I20260620 14:47:29.118047 2580 raft_consensus.cc:2454] T 6121447adcd045d7b4d20c85cd37d891 P 7f1b44e3bb694105afe8822fc7cb348b [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e5717b124bc7489e8817304e3108a0eb in term 1.
I20260620 14:47:29.118507 2373 leader_election.cc:302] T 6121447adcd045d7b4d20c85cd37d891 P e5717b124bc7489e8817304e3108a0eb [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5aa58728f3b74083b017cdcd573bc626, e5717b124bc7489e8817304e3108a0eb; no voters:
I20260620 14:47:29.118863 2744 raft_consensus.cc:2781] T 6121447adcd045d7b4d20c85cd37d891 P e5717b124bc7489e8817304e3108a0eb [term 1 FOLLOWER]: Leader election won for term 1
I20260620 14:47:29.119380 2744 raft_consensus.cc:686] T 6121447adcd045d7b4d20c85cd37d891 P e5717b124bc7489e8817304e3108a0eb [term 1 LEADER]: Becoming Leader. State: Replica: e5717b124bc7489e8817304e3108a0eb, State: Running, Role: LEADER
I20260620 14:47:29.119727 2744 consensus_queue.cc:237] T 6121447adcd045d7b4d20c85cd37d891 P e5717b124bc7489e8817304e3108a0eb [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5aa58728f3b74083b017cdcd573bc626" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 35927 } } peers { permanent_uuid: "e5717b124bc7489e8817304e3108a0eb" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 37155 } } peers { permanent_uuid: "7f1b44e3bb694105afe8822fc7cb348b" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 32947 } }
I20260620 14:47:29.122367 2142 catalog_manager.cc:5495] T 6121447adcd045d7b4d20c85cd37d891 P e5717b124bc7489e8817304e3108a0eb reported cstate change: term changed from 0 to 1, leader changed from <none> to e5717b124bc7489e8817304e3108a0eb (127.0.97.2). New cstate: current_term: 1 leader_uuid: "e5717b124bc7489e8817304e3108a0eb" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5aa58728f3b74083b017cdcd573bc626" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 35927 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "e5717b124bc7489e8817304e3108a0eb" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 37155 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "7f1b44e3bb694105afe8822fc7cb348b" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 32947 } health_report { overall_health: UNKNOWN } } }
I20260620 14:47:29.148558 2749 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:47:29.143203 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41760 (local address 127.0.97.2:37155)
0620 14:47:29.144283 (+ 1080us) server_negotiation.cc:206] Beginning negotiation
0620 14:47:29.144287 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 14:47:29.144314 (+ 27us) server_negotiation.cc:407] Connection header received
0620 14:47:29.144370 (+ 56us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:47:29.144373 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:47:29.144433 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:47:29.144516 (+ 83us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 14:47:29.144865 (+ 349us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:47:29.145631 (+ 766us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:47:29.146911 (+ 1280us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:47:29.147337 (+ 426us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:47:29.147426 (+ 89us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:47:29.147831 (+ 405us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 14:47:29.147938 (+ 107us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0620 14:47:29.148003 (+ 65us) server_negotiation.cc:1035] Waiting for connection context
0620 14:47:29.148123 (+ 120us) server_negotiation.cc:299] Negotiation successful
0620 14:47:29.148185 (+ 62us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":963,"thread_start_us":845,"threads_started":1}
I20260620 14:47:29.157657 2301 raft_consensus.cc:1261] T 6121447adcd045d7b4d20c85cd37d891 P 5aa58728f3b74083b017cdcd573bc626 [term 1 FOLLOWER]: Refusing update from remote peer e5717b124bc7489e8817304e3108a0eb: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 14:47:29.157653 2580 raft_consensus.cc:1261] T 6121447adcd045d7b4d20c85cd37d891 P 7f1b44e3bb694105afe8822fc7cb348b [term 1 FOLLOWER]: Refusing update from remote peer e5717b124bc7489e8817304e3108a0eb: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 14:47:29.158179 2746 consensus_queue.cc:1035] T 6121447adcd045d7b4d20c85cd37d891 P e5717b124bc7489e8817304e3108a0eb [LEADER]: Connected to new peer: Peer: permanent_uuid: "5aa58728f3b74083b017cdcd573bc626" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 35927 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 14:47:29.158506 2744 consensus_queue.cc:1035] T 6121447adcd045d7b4d20c85cd37d891 P e5717b124bc7489e8817304e3108a0eb [LEADER]: Connected to new peer: Peer: permanent_uuid: "7f1b44e3bb694105afe8822fc7cb348b" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 32947 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 14:47:29.199930 2142 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:40420:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 14:47:29.200174 2142 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:40420:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 14:47:29.202790 2142 catalog_manager.cc:5782] T 00000000000000000000000000000000 P dbf70d79ae0b41f28a6637ed9ab0ecc4: Sending DeleteTablet for 3 replicas of tablet 6121447adcd045d7b4d20c85cd37d891
I20260620 14:47:29.203524 2281 tablet_service.cc:1514] Processing DeleteTablet for tablet 6121447adcd045d7b4d20c85cd37d891 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 14:47:29 UTC) from {username='kudu', principal='kudu/127.0.97.62@KRBTEST.COM'} at 127.0.0.1:48790
I20260620 14:47:29.203846 2727 tablet_replica.cc:330] T 6121447adcd045d7b4d20c85cd37d891 P 5aa58728f3b74083b017cdcd573bc626: stopping tablet replica
I20260620 14:47:29.204051 2727 raft_consensus.cc:2229] T 6121447adcd045d7b4d20c85cd37d891 P 5aa58728f3b74083b017cdcd573bc626 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 14:47:29.204119 2560 tablet_service.cc:1514] Processing DeleteTablet for tablet 6121447adcd045d7b4d20c85cd37d891 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 14:47:29 UTC) from {username='kudu', principal='kudu/127.0.97.62@KRBTEST.COM'} at 127.0.0.1:46328
I20260620 14:47:29.204248 2727 raft_consensus.cc:2258] T 6121447adcd045d7b4d20c85cd37d891 P 5aa58728f3b74083b017cdcd573bc626 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 14:47:29.204386 2729 tablet_replica.cc:330] T 6121447adcd045d7b4d20c85cd37d891 P 7f1b44e3bb694105afe8822fc7cb348b: stopping tablet replica
I20260620 14:47:29.204524 2420 tablet_service.cc:1514] Processing DeleteTablet for tablet 6121447adcd045d7b4d20c85cd37d891 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 14:47:29 UTC) from {username='kudu', principal='kudu/127.0.97.62@KRBTEST.COM'} at 127.0.0.1:41730
I20260620 14:47:29.204574 2729 raft_consensus.cc:2229] T 6121447adcd045d7b4d20c85cd37d891 P 7f1b44e3bb694105afe8822fc7cb348b [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 14:47:29.204752 2729 raft_consensus.cc:2258] T 6121447adcd045d7b4d20c85cd37d891 P 7f1b44e3bb694105afe8822fc7cb348b [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 14:47:29.204782 2728 tablet_replica.cc:330] T 6121447adcd045d7b4d20c85cd37d891 P e5717b124bc7489e8817304e3108a0eb: stopping tablet replica
I20260620 14:47:29.204978 2728 raft_consensus.cc:2229] T 6121447adcd045d7b4d20c85cd37d891 P e5717b124bc7489e8817304e3108a0eb [term 1 LEADER]: Raft consensus shutting down.
I20260620 14:47:29.205108 2142 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:40420:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20260620 14:47:29.205271 2728 raft_consensus.cc:2258] T 6121447adcd045d7b4d20c85cd37d891 P e5717b124bc7489e8817304e3108a0eb [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 14:47:29.205487 2727 ts_tablet_manager.cc:1905] T 6121447adcd045d7b4d20c85cd37d891 P 5aa58728f3b74083b017cdcd573bc626: Deleting tablet data with delete state TABLET_DATA_DELETED
W20260620 14:47:29.205639 2142 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260620 14:47:29.206369 2728 ts_tablet_manager.cc:1905] T 6121447adcd045d7b4d20c85cd37d891 P e5717b124bc7489e8817304e3108a0eb: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 14:47:29.207743 2729 ts_tablet_manager.cc:1905] T 6121447adcd045d7b4d20c85cd37d891 P 7f1b44e3bb694105afe8822fc7cb348b: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 14:47:29.212323 2727 ts_tablet_manager.cc:1918] T 6121447adcd045d7b4d20c85cd37d891 P 5aa58728f3b74083b017cdcd573bc626: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260620 14:47:29.212479 2727 log.cc:1196] T 6121447adcd045d7b4d20c85cd37d891 P 5aa58728f3b74083b017cdcd573bc626: Deleting WAL directory at /tmp/dist-test-taskUci3DU/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781966829091300-388-0/minicluster-data/ts-0/wal/wals/6121447adcd045d7b4d20c85cd37d891
I20260620 14:47:29.212893 2727 ts_tablet_manager.cc:1939] T 6121447adcd045d7b4d20c85cd37d891 P 5aa58728f3b74083b017cdcd573bc626: Deleting consensus metadata
I20260620 14:47:29.212705 2729 ts_tablet_manager.cc:1918] T 6121447adcd045d7b4d20c85cd37d891 P 7f1b44e3bb694105afe8822fc7cb348b: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260620 14:47:29.213049 2729 log.cc:1196] T 6121447adcd045d7b4d20c85cd37d891 P 7f1b44e3bb694105afe8822fc7cb348b: Deleting WAL directory at /tmp/dist-test-taskUci3DU/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781966829091300-388-0/minicluster-data/ts-2/wal/wals/6121447adcd045d7b4d20c85cd37d891
I20260620 14:47:29.213557 2728 ts_tablet_manager.cc:1918] T 6121447adcd045d7b4d20c85cd37d891 P e5717b124bc7489e8817304e3108a0eb: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260620 14:47:29.213789 2728 log.cc:1196] T 6121447adcd045d7b4d20c85cd37d891 P e5717b124bc7489e8817304e3108a0eb: Deleting WAL directory at /tmp/dist-test-taskUci3DU/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781966829091300-388-0/minicluster-data/ts-1/wal/wals/6121447adcd045d7b4d20c85cd37d891
I20260620 14:47:29.213970 2130 catalog_manager.cc:4841] TS 5aa58728f3b74083b017cdcd573bc626 (127.0.97.1:35927): tablet 6121447adcd045d7b4d20c85cd37d891 (table test-table [id=72c70197be594173b771df013c32267f]) successfully deleted
I20260620 14:47:29.214411 2728 ts_tablet_manager.cc:1939] T 6121447adcd045d7b4d20c85cd37d891 P e5717b124bc7489e8817304e3108a0eb: Deleting consensus metadata
I20260620 14:47:29.214604 2729 ts_tablet_manager.cc:1939] T 6121447adcd045d7b4d20c85cd37d891 P 7f1b44e3bb694105afe8822fc7cb348b: Deleting consensus metadata
I20260620 14:47:29.215587 2130 catalog_manager.cc:4841] TS 7f1b44e3bb694105afe8822fc7cb348b (127.0.97.3:32947): tablet 6121447adcd045d7b4d20c85cd37d891 (table test-table [id=72c70197be594173b771df013c32267f]) successfully deleted
I20260620 14:47:29.215929 2129 catalog_manager.cc:4841] TS e5717b124bc7489e8817304e3108a0eb (127.0.97.2:37155): tablet 6121447adcd045d7b4d20c85cd37d891 (table test-table [id=72c70197be594173b771df013c32267f]) successfully deleted
I20260620 14:47:29.216773 2281 tablet_service.cc:1467] Processing CreateTablet for tablet 86b00e18f13b4b9eb9b39010f3e6f17f (DEFAULT_TABLE table=test-table [id=4e5193bde999478bae94c03557749add]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 14:47:29.216962 2420 tablet_service.cc:1467] Processing CreateTablet for tablet 86b00e18f13b4b9eb9b39010f3e6f17f (DEFAULT_TABLE table=test-table [id=4e5193bde999478bae94c03557749add]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 14:47:29.217211 2281 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 86b00e18f13b4b9eb9b39010f3e6f17f. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 14:47:29.217314 2420 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 86b00e18f13b4b9eb9b39010f3e6f17f. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 14:47:29.217406 2560 tablet_service.cc:1467] Processing CreateTablet for tablet 86b00e18f13b4b9eb9b39010f3e6f17f (DEFAULT_TABLE table=test-table [id=4e5193bde999478bae94c03557749add]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 14:47:29.217782 2560 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 86b00e18f13b4b9eb9b39010f3e6f17f. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 14:47:29.222929 2738 tablet_bootstrap.cc:492] T 86b00e18f13b4b9eb9b39010f3e6f17f P 7f1b44e3bb694105afe8822fc7cb348b: Bootstrap starting.
I20260620 14:47:29.223652 2739 tablet_bootstrap.cc:492] T 86b00e18f13b4b9eb9b39010f3e6f17f P e5717b124bc7489e8817304e3108a0eb: Bootstrap starting.
I20260620 14:47:29.224509 2740 tablet_bootstrap.cc:492] T 86b00e18f13b4b9eb9b39010f3e6f17f P 5aa58728f3b74083b017cdcd573bc626: Bootstrap starting.
I20260620 14:47:29.224592 2738 tablet_bootstrap.cc:654] T 86b00e18f13b4b9eb9b39010f3e6f17f P 7f1b44e3bb694105afe8822fc7cb348b: Neither blocks nor log segments found. Creating new log.
I20260620 14:47:29.225208 2739 tablet_bootstrap.cc:654] T 86b00e18f13b4b9eb9b39010f3e6f17f P e5717b124bc7489e8817304e3108a0eb: Neither blocks nor log segments found. Creating new log.
I20260620 14:47:29.226135 2740 tablet_bootstrap.cc:654] T 86b00e18f13b4b9eb9b39010f3e6f17f P 5aa58728f3b74083b017cdcd573bc626: Neither blocks nor log segments found. Creating new log.
I20260620 14:47:29.226596 2738 tablet_bootstrap.cc:492] T 86b00e18f13b4b9eb9b39010f3e6f17f P 7f1b44e3bb694105afe8822fc7cb348b: No bootstrap required, opened a new log
I20260620 14:47:29.226776 2738 ts_tablet_manager.cc:1397] T 86b00e18f13b4b9eb9b39010f3e6f17f P 7f1b44e3bb694105afe8822fc7cb348b: Time spent bootstrapping tablet: real 0.004s user 0.003s sys 0.000s
I20260620 14:47:29.227622 2739 tablet_bootstrap.cc:492] T 86b00e18f13b4b9eb9b39010f3e6f17f P e5717b124bc7489e8817304e3108a0eb: No bootstrap required, opened a new log
I20260620 14:47:29.227831 2739 ts_tablet_manager.cc:1397] T 86b00e18f13b4b9eb9b39010f3e6f17f P e5717b124bc7489e8817304e3108a0eb: Time spent bootstrapping tablet: real 0.004s user 0.003s sys 0.000s
I20260620 14:47:29.228065 2738 raft_consensus.cc:348] T 86b00e18f13b4b9eb9b39010f3e6f17f P 7f1b44e3bb694105afe8822fc7cb348b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5aa58728f3b74083b017cdcd573bc626" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 35927 } } peers { permanent_uuid: "e5717b124bc7489e8817304e3108a0eb" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 37155 } } peers { permanent_uuid: "7f1b44e3bb694105afe8822fc7cb348b" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 32947 } }
I20260620 14:47:29.228315 2738 raft_consensus.cc:374] T 86b00e18f13b4b9eb9b39010f3e6f17f P 7f1b44e3bb694105afe8822fc7cb348b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 14:47:29.228412 2738 raft_consensus.cc:729] T 86b00e18f13b4b9eb9b39010f3e6f17f P 7f1b44e3bb694105afe8822fc7cb348b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7f1b44e3bb694105afe8822fc7cb348b, State: Initialized, Role: FOLLOWER
I20260620 14:47:29.228703 2738 consensus_queue.cc:260] T 86b00e18f13b4b9eb9b39010f3e6f17f P 7f1b44e3bb694105afe8822fc7cb348b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5aa58728f3b74083b017cdcd573bc626" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 35927 } } peers { permanent_uuid: "e5717b124bc7489e8817304e3108a0eb" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 37155 } } peers { permanent_uuid: "7f1b44e3bb694105afe8822fc7cb348b" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 32947 } }
I20260620 14:47:29.228716 2739 raft_consensus.cc:348] T 86b00e18f13b4b9eb9b39010f3e6f17f P e5717b124bc7489e8817304e3108a0eb [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5aa58728f3b74083b017cdcd573bc626" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 35927 } } peers { permanent_uuid: "e5717b124bc7489e8817304e3108a0eb" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 37155 } } peers { permanent_uuid: "7f1b44e3bb694105afe8822fc7cb348b" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 32947 } }
I20260620 14:47:29.228978 2739 raft_consensus.cc:374] T 86b00e18f13b4b9eb9b39010f3e6f17f P e5717b124bc7489e8817304e3108a0eb [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 14:47:29.229058 2739 raft_consensus.cc:729] T 86b00e18f13b4b9eb9b39010f3e6f17f P e5717b124bc7489e8817304e3108a0eb [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e5717b124bc7489e8817304e3108a0eb, State: Initialized, Role: FOLLOWER
I20260620 14:47:29.229310 2738 ts_tablet_manager.cc:1428] T 86b00e18f13b4b9eb9b39010f3e6f17f P 7f1b44e3bb694105afe8822fc7cb348b: Time spent starting tablet: real 0.002s user 0.000s sys 0.003s
I20260620 14:47:29.229283 2739 consensus_queue.cc:260] T 86b00e18f13b4b9eb9b39010f3e6f17f P e5717b124bc7489e8817304e3108a0eb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5aa58728f3b74083b017cdcd573bc626" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 35927 } } peers { permanent_uuid: "e5717b124bc7489e8817304e3108a0eb" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 37155 } } peers { permanent_uuid: "7f1b44e3bb694105afe8822fc7cb348b" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 32947 } }
I20260620 14:47:29.229678 2740 tablet_bootstrap.cc:492] T 86b00e18f13b4b9eb9b39010f3e6f17f P 5aa58728f3b74083b017cdcd573bc626: No bootstrap required, opened a new log
I20260620 14:47:29.229786 2739 ts_tablet_manager.cc:1428] T 86b00e18f13b4b9eb9b39010f3e6f17f P e5717b124bc7489e8817304e3108a0eb: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20260620 14:47:29.229830 2740 ts_tablet_manager.cc:1397] T 86b00e18f13b4b9eb9b39010f3e6f17f P 5aa58728f3b74083b017cdcd573bc626: Time spent bootstrapping tablet: real 0.005s user 0.000s sys 0.004s
I20260620 14:47:29.230721 2740 raft_consensus.cc:348] T 86b00e18f13b4b9eb9b39010f3e6f17f P 5aa58728f3b74083b017cdcd573bc626 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5aa58728f3b74083b017cdcd573bc626" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 35927 } } peers { permanent_uuid: "e5717b124bc7489e8817304e3108a0eb" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 37155 } } peers { permanent_uuid: "7f1b44e3bb694105afe8822fc7cb348b" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 32947 } }
I20260620 14:47:29.230968 2740 raft_consensus.cc:374] T 86b00e18f13b4b9eb9b39010f3e6f17f P 5aa58728f3b74083b017cdcd573bc626 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 14:47:29.231056 2740 raft_consensus.cc:729] T 86b00e18f13b4b9eb9b39010f3e6f17f P 5aa58728f3b74083b017cdcd573bc626 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5aa58728f3b74083b017cdcd573bc626, State: Initialized, Role: FOLLOWER
I20260620 14:47:29.231287 2740 consensus_queue.cc:260] T 86b00e18f13b4b9eb9b39010f3e6f17f P 5aa58728f3b74083b017cdcd573bc626 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5aa58728f3b74083b017cdcd573bc626" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 35927 } } peers { permanent_uuid: "e5717b124bc7489e8817304e3108a0eb" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 37155 } } peers { permanent_uuid: "7f1b44e3bb694105afe8822fc7cb348b" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 32947 } }
I20260620 14:47:29.231845 2740 ts_tablet_manager.cc:1428] T 86b00e18f13b4b9eb9b39010f3e6f17f P 5aa58728f3b74083b017cdcd573bc626: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20260620 14:47:29.285075 2745 raft_consensus.cc:482] T 86b00e18f13b4b9eb9b39010f3e6f17f P 5aa58728f3b74083b017cdcd573bc626 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260620 14:47:29.285341 2745 raft_consensus.cc:504] T 86b00e18f13b4b9eb9b39010f3e6f17f P 5aa58728f3b74083b017cdcd573bc626 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5aa58728f3b74083b017cdcd573bc626" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 35927 } } peers { permanent_uuid: "e5717b124bc7489e8817304e3108a0eb" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 37155 } } peers { permanent_uuid: "7f1b44e3bb694105afe8822fc7cb348b" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 32947 } }
I20260620 14:47:29.287051 2745 leader_election.cc:288] T 86b00e18f13b4b9eb9b39010f3e6f17f P 5aa58728f3b74083b017cdcd573bc626 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers e5717b124bc7489e8817304e3108a0eb (127.0.97.2:37155), 7f1b44e3bb694105afe8822fc7cb348b (127.0.97.3:32947)
I20260620 14:47:29.294699 2749 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:47:29.287686 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.1:56787 (local address 127.0.97.2:37155)
0620 14:47:29.288213 (+ 527us) server_negotiation.cc:206] Beginning negotiation
0620 14:47:29.288220 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0620 14:47:29.288686 (+ 466us) server_negotiation.cc:407] Connection header received
0620 14:47:29.288995 (+ 309us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:47:29.289000 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:47:29.289076 (+ 76us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:47:29.289141 (+ 65us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 14:47:29.289947 (+ 806us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:47:29.290683 (+ 736us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:47:29.292999 (+ 2316us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:47:29.293769 (+ 770us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:47:29.293899 (+ 130us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:47:29.294023 (+ 124us) server_negotiation.cc:1035] Waiting for connection context
0620 14:47:29.294321 (+ 298us) server_negotiation.cc:299] Negotiation successful
0620 14:47:29.294446 (+ 125us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":365}
I20260620 14:47:29.294843 2760 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:47:29.287888 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.1:58683 (local address 127.0.97.3:32947)
0620 14:47:29.289167 (+ 1279us) server_negotiation.cc:206] Beginning negotiation
0620 14:47:29.289173 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 14:47:29.289195 (+ 22us) server_negotiation.cc:407] Connection header received
0620 14:47:29.289259 (+ 64us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:47:29.289265 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:47:29.289344 (+ 79us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:47:29.289444 (+ 100us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 14:47:29.290055 (+ 611us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:47:29.290833 (+ 778us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:47:29.293052 (+ 2219us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:47:29.293799 (+ 747us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:47:29.293955 (+ 156us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:47:29.294060 (+ 105us) server_negotiation.cc:1035] Waiting for connection context
0620 14:47:29.294364 (+ 304us) server_negotiation.cc:299] Negotiation successful
0620 14:47:29.294495 (+ 131us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1127,"thread_start_us":997,"threads_started":1}
I20260620 14:47:29.294896 2759 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:47:29.287517 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.3:32947 (local address 127.0.97.1:58683)
0620 14:47:29.288944 (+ 1427us) negotiation.cc:107] Waiting for socket to connect
0620 14:47:29.288961 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0620 14:47:29.289070 (+ 109us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 14:47:29.289463 (+ 393us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 14:47:29.289465 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 14:47:29.289491 (+ 26us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 14:47:29.289850 (+ 359us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:47:29.289883 (+ 33us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:47:29.290961 (+ 1078us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:47:29.290964 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:47:29.292863 (+ 1899us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:47:29.292875 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:47:29.294044 (+ 1169us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:47:29.294052 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:47:29.294184 (+ 132us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:47:29.294205 (+ 21us) client_negotiation.cc:769] Sending connection context
0620 14:47:29.294310 (+ 105us) client_negotiation.cc:240] Negotiation successful
0620 14:47:29.294436 (+ 126us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1275,"mutex_wait_us":24,"thread_start_us":1030,"threads_started":1}
I20260620 14:47:29.295275 2758 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:47:29.287273 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.2:37155 (local address 127.0.97.1:56787)
0620 14:47:29.288567 (+ 1294us) negotiation.cc:107] Waiting for socket to connect
0620 14:47:29.288590 (+ 23us) client_negotiation.cc:174] Beginning negotiation
0620 14:47:29.288807 (+ 217us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 14:47:29.289362 (+ 555us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 14:47:29.289369 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 14:47:29.289398 (+ 29us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 14:47:29.289766 (+ 368us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:47:29.289781 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:47:29.290833 (+ 1052us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:47:29.290839 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:47:29.292844 (+ 2005us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:47:29.292856 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:47:29.293985 (+ 1129us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:47:29.293992 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:47:29.294159 (+ 167us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:47:29.294186 (+ 27us) client_negotiation.cc:769] Sending connection context
0620 14:47:29.294267 (+ 81us) client_negotiation.cc:240] Negotiation successful
0620 14:47:29.294415 (+ 148us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1107,"mutex_wait_us":56,"thread_start_us":898,"threads_started":1}
I20260620 14:47:29.295538 2580 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "86b00e18f13b4b9eb9b39010f3e6f17f" candidate_uuid: "5aa58728f3b74083b017cdcd573bc626" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7f1b44e3bb694105afe8822fc7cb348b" is_pre_election: true
I20260620 14:47:29.295823 2580 raft_consensus.cc:2454] T 86b00e18f13b4b9eb9b39010f3e6f17f P 7f1b44e3bb694105afe8822fc7cb348b [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5aa58728f3b74083b017cdcd573bc626 in term 0.
I20260620 14:47:29.295915 2441 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "86b00e18f13b4b9eb9b39010f3e6f17f" candidate_uuid: "5aa58728f3b74083b017cdcd573bc626" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e5717b124bc7489e8817304e3108a0eb" is_pre_election: true
I20260620 14:47:29.296154 2441 raft_consensus.cc:2454] T 86b00e18f13b4b9eb9b39010f3e6f17f P e5717b124bc7489e8817304e3108a0eb [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5aa58728f3b74083b017cdcd573bc626 in term 0.
I20260620 14:47:29.296289 2233 leader_election.cc:302] T 86b00e18f13b4b9eb9b39010f3e6f17f P 5aa58728f3b74083b017cdcd573bc626 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5aa58728f3b74083b017cdcd573bc626, 7f1b44e3bb694105afe8822fc7cb348b; no voters:
I20260620 14:47:29.296660 2745 raft_consensus.cc:2781] T 86b00e18f13b4b9eb9b39010f3e6f17f P 5aa58728f3b74083b017cdcd573bc626 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260620 14:47:29.296794 2745 raft_consensus.cc:482] T 86b00e18f13b4b9eb9b39010f3e6f17f P 5aa58728f3b74083b017cdcd573bc626 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260620 14:47:29.296908 2745 raft_consensus.cc:3037] T 86b00e18f13b4b9eb9b39010f3e6f17f P 5aa58728f3b74083b017cdcd573bc626 [term 0 FOLLOWER]: Advancing to term 1
I20260620 14:47:29.299000 2745 raft_consensus.cc:504] T 86b00e18f13b4b9eb9b39010f3e6f17f P 5aa58728f3b74083b017cdcd573bc626 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5aa58728f3b74083b017cdcd573bc626" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 35927 } } peers { permanent_uuid: "e5717b124bc7489e8817304e3108a0eb" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 37155 } } peers { permanent_uuid: "7f1b44e3bb694105afe8822fc7cb348b" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 32947 } }
I20260620 14:47:29.299638 2745 leader_election.cc:288] T 86b00e18f13b4b9eb9b39010f3e6f17f P 5aa58728f3b74083b017cdcd573bc626 [CANDIDATE]: Term 1 election: Requested vote from peers e5717b124bc7489e8817304e3108a0eb (127.0.97.2:37155), 7f1b44e3bb694105afe8822fc7cb348b (127.0.97.3:32947)
I20260620 14:47:29.300141 2580 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "86b00e18f13b4b9eb9b39010f3e6f17f" candidate_uuid: "5aa58728f3b74083b017cdcd573bc626" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7f1b44e3bb694105afe8822fc7cb348b"
I20260620 14:47:29.300129 2441 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "86b00e18f13b4b9eb9b39010f3e6f17f" candidate_uuid: "5aa58728f3b74083b017cdcd573bc626" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e5717b124bc7489e8817304e3108a0eb"
I20260620 14:47:29.300379 2441 raft_consensus.cc:3037] T 86b00e18f13b4b9eb9b39010f3e6f17f P e5717b124bc7489e8817304e3108a0eb [term 0 FOLLOWER]: Advancing to term 1
I20260620 14:47:29.300379 2580 raft_consensus.cc:3037] T 86b00e18f13b4b9eb9b39010f3e6f17f P 7f1b44e3bb694105afe8822fc7cb348b [term 0 FOLLOWER]: Advancing to term 1
I20260620 14:47:29.303296 2441 raft_consensus.cc:2454] T 86b00e18f13b4b9eb9b39010f3e6f17f P e5717b124bc7489e8817304e3108a0eb [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5aa58728f3b74083b017cdcd573bc626 in term 1.
I20260620 14:47:29.303300 2580 raft_consensus.cc:2454] T 86b00e18f13b4b9eb9b39010f3e6f17f P 7f1b44e3bb694105afe8822fc7cb348b [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5aa58728f3b74083b017cdcd573bc626 in term 1.
I20260620 14:47:29.303838 2233 leader_election.cc:302] T 86b00e18f13b4b9eb9b39010f3e6f17f P 5aa58728f3b74083b017cdcd573bc626 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5aa58728f3b74083b017cdcd573bc626, 7f1b44e3bb694105afe8822fc7cb348b; no voters:
I20260620 14:47:29.304152 2745 raft_consensus.cc:2781] T 86b00e18f13b4b9eb9b39010f3e6f17f P 5aa58728f3b74083b017cdcd573bc626 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 14:47:29.304947 2745 raft_consensus.cc:686] T 86b00e18f13b4b9eb9b39010f3e6f17f P 5aa58728f3b74083b017cdcd573bc626 [term 1 LEADER]: Becoming Leader. State: Replica: 5aa58728f3b74083b017cdcd573bc626, State: Running, Role: LEADER
I20260620 14:47:29.305322 2745 consensus_queue.cc:237] T 86b00e18f13b4b9eb9b39010f3e6f17f P 5aa58728f3b74083b017cdcd573bc626 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5aa58728f3b74083b017cdcd573bc626" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 35927 } } peers { permanent_uuid: "e5717b124bc7489e8817304e3108a0eb" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 37155 } } peers { permanent_uuid: "7f1b44e3bb694105afe8822fc7cb348b" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 32947 } }
I20260620 14:47:29.308327 2144 catalog_manager.cc:5495] T 86b00e18f13b4b9eb9b39010f3e6f17f P 5aa58728f3b74083b017cdcd573bc626 reported cstate change: term changed from 0 to 1, leader changed from <none> to 5aa58728f3b74083b017cdcd573bc626 (127.0.97.1). New cstate: current_term: 1 leader_uuid: "5aa58728f3b74083b017cdcd573bc626" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5aa58728f3b74083b017cdcd573bc626" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 35927 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "e5717b124bc7489e8817304e3108a0eb" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 37155 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "7f1b44e3bb694105afe8822fc7cb348b" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 32947 } health_report { overall_health: UNKNOWN } } }
I20260620 14:47:29.339885 2763 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:47:29.334249 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48792 (local address 127.0.97.1:35927)
0620 14:47:29.334794 (+ 545us) server_negotiation.cc:206] Beginning negotiation
0620 14:47:29.334798 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 14:47:29.334817 (+ 19us) server_negotiation.cc:407] Connection header received
0620 14:47:29.334893 (+ 76us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:47:29.334896 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:47:29.334961 (+ 65us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:47:29.335099 (+ 138us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 14:47:29.335473 (+ 374us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:47:29.336320 (+ 847us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:47:29.337677 (+ 1357us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:47:29.338117 (+ 440us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:47:29.338229 (+ 112us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:47:29.338738 (+ 509us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 14:47:29.339319 (+ 581us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0620 14:47:29.339425 (+ 106us) server_negotiation.cc:1035] Waiting for connection context
0620 14:47:29.339526 (+ 101us) server_negotiation.cc:299] Negotiation successful
0620 14:47:29.339591 (+ 65us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":455,"thread_start_us":340,"threads_started":1}
I20260620 14:47:29.354561 2760 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:47:29.349721 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.1:58103 (local address 127.0.97.3:32947)
0620 14:47:29.349884 (+ 163us) server_negotiation.cc:206] Beginning negotiation
0620 14:47:29.349888 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 14:47:29.350327 (+ 439us) server_negotiation.cc:407] Connection header received
0620 14:47:29.350484 (+ 157us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:47:29.350488 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:47:29.350552 (+ 64us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:47:29.350619 (+ 67us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 14:47:29.351102 (+ 483us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:47:29.351924 (+ 822us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:47:29.353217 (+ 1293us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:47:29.353601 (+ 384us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:47:29.353692 (+ 91us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:47:29.354042 (+ 350us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 14:47:29.354133 (+ 91us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0620 14:47:29.354194 (+ 61us) server_negotiation.cc:1035] Waiting for connection context
0620 14:47:29.354315 (+ 121us) server_negotiation.cc:299] Negotiation successful
0620 14:47:29.354375 (+ 60us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":50}
I20260620 14:47:29.354650 2766 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:47:29.349560 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.3:32947 (local address 127.0.97.1:58103)
0620 14:47:29.350231 (+ 671us) negotiation.cc:107] Waiting for socket to connect
0620 14:47:29.350247 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0620 14:47:29.350367 (+ 120us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 14:47:29.350641 (+ 274us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 14:47:29.350644 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 14:47:29.350653 (+ 9us) client_negotiation.cc:189] Negotiated authn=TOKEN
0620 14:47:29.350972 (+ 319us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:47:29.350980 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:47:29.352064 (+ 1084us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:47:29.352067 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:47:29.353098 (+ 1031us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:47:29.353109 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:47:29.353737 (+ 628us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:47:29.353741 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:47:29.353854 (+ 113us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:47:29.353910 (+ 56us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0620 14:47:29.354207 (+ 297us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0620 14:47:29.354214 (+ 7us) client_negotiation.cc:769] Sending connection context
0620 14:47:29.354273 (+ 59us) client_negotiation.cc:240] Negotiation successful
0620 14:47:29.354367 (+ 94us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":514,"thread_start_us":323,"threads_started":1}
I20260620 14:47:29.362675 2580 raft_consensus.cc:1261] T 86b00e18f13b4b9eb9b39010f3e6f17f P 7f1b44e3bb694105afe8822fc7cb348b [term 1 FOLLOWER]: Refusing update from remote peer 5aa58728f3b74083b017cdcd573bc626: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 14:47:29.362675 2441 raft_consensus.cc:1261] T 86b00e18f13b4b9eb9b39010f3e6f17f P e5717b124bc7489e8817304e3108a0eb [term 1 FOLLOWER]: Refusing update from remote peer 5aa58728f3b74083b017cdcd573bc626: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 14:47:29.363358 2745 consensus_queue.cc:1035] T 86b00e18f13b4b9eb9b39010f3e6f17f P 5aa58728f3b74083b017cdcd573bc626 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7f1b44e3bb694105afe8822fc7cb348b" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 32947 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 14:47:29.363735 2761 consensus_queue.cc:1035] T 86b00e18f13b4b9eb9b39010f3e6f17f P 5aa58728f3b74083b017cdcd573bc626 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e5717b124bc7489e8817304e3108a0eb" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 37155 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 14:47:29.368300 2712 mvcc.cc:205] Tried to move back new op lower bound from 7298936214983741440 to 7298936214757814272. Current Snapshot: MvccSnapshot[applied={T|T < 7298936214983741440}]
I20260620 14:47:29.390861 2763 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:47:29.385395 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.3:48151 (local address 127.0.97.1:35927)
0620 14:47:29.385552 (+ 157us) server_negotiation.cc:206] Beginning negotiation
0620 14:47:29.385561 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0620 14:47:29.386012 (+ 451us) server_negotiation.cc:407] Connection header received
0620 14:47:29.386268 (+ 256us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:47:29.386273 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:47:29.386359 (+ 86us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:47:29.386439 (+ 80us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 14:47:29.386855 (+ 416us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:47:29.387522 (+ 667us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:47:29.389196 (+ 1674us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:47:29.389823 (+ 627us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:47:29.389922 (+ 99us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:47:29.390326 (+ 404us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 14:47:29.390427 (+ 101us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0620 14:47:29.390497 (+ 70us) server_negotiation.cc:1035] Waiting for connection context
0620 14:47:29.390588 (+ 91us) server_negotiation.cc:299] Negotiation successful
0620 14:47:29.390646 (+ 58us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":31}
I20260620 14:47:29.390926 2770 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:47:29.385317 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:35927 (local address 127.0.97.3:48151)
0620 14:47:29.385917 (+ 600us) negotiation.cc:107] Waiting for socket to connect
0620 14:47:29.385936 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0620 14:47:29.386078 (+ 142us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 14:47:29.386483 (+ 405us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 14:47:29.386487 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 14:47:29.386500 (+ 13us) client_negotiation.cc:189] Negotiated authn=TOKEN
0620 14:47:29.386732 (+ 232us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:47:29.386740 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:47:29.387692 (+ 952us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:47:29.387697 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:47:29.389031 (+ 1334us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:47:29.389041 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:47:29.389976 (+ 935us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:47:29.389981 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:47:29.390098 (+ 117us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:47:29.390153 (+ 55us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0620 14:47:29.390509 (+ 356us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0620 14:47:29.390517 (+ 8us) client_negotiation.cc:769] Sending connection context
0620 14:47:29.390576 (+ 59us) client_negotiation.cc:240] Negotiation successful
0620 14:47:29.390677 (+ 101us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":460,"thread_start_us":306,"threads_started":1}
W20260620 14:47:29.391953 2712 tablet_replica.cc:1298] Aborted: operation has been aborted: cancelling pending write operations
I20260620 14:47:31.005884 2144 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:40420:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 14:47:31.006122 2144 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:40420:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 14:47:31.009939 2144 catalog_manager.cc:5782] T 00000000000000000000000000000000 P dbf70d79ae0b41f28a6637ed9ab0ecc4: Sending DeleteTablet for 3 replicas of tablet 86b00e18f13b4b9eb9b39010f3e6f17f
I20260620 14:47:31.010713 2281 tablet_service.cc:1514] Processing DeleteTablet for tablet 86b00e18f13b4b9eb9b39010f3e6f17f with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 14:47:31 UTC) from {username='kudu', principal='kudu/127.0.97.62@KRBTEST.COM'} at 127.0.0.1:48790
I20260620 14:47:31.010857 2420 tablet_service.cc:1514] Processing DeleteTablet for tablet 86b00e18f13b4b9eb9b39010f3e6f17f with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 14:47:31 UTC) from {username='kudu', principal='kudu/127.0.97.62@KRBTEST.COM'} at 127.0.0.1:41730
I20260620 14:47:31.010936 2560 tablet_service.cc:1514] Processing DeleteTablet for tablet 86b00e18f13b4b9eb9b39010f3e6f17f with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 14:47:31 UTC) from {username='kudu', principal='kudu/127.0.97.62@KRBTEST.COM'} at 127.0.0.1:46328
I20260620 14:47:31.011700 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskUci3DU/build/asan/bin/kudu pid 2216
I20260620 14:47:31.024187 2796 tablet_replica.cc:330] T 86b00e18f13b4b9eb9b39010f3e6f17f P 7f1b44e3bb694105afe8822fc7cb348b: stopping tablet replica
I20260620 14:47:31.024477 2796 raft_consensus.cc:2229] T 86b00e18f13b4b9eb9b39010f3e6f17f P 7f1b44e3bb694105afe8822fc7cb348b [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 14:47:31.024760 2796 raft_consensus.cc:2258] T 86b00e18f13b4b9eb9b39010f3e6f17f P 7f1b44e3bb694105afe8822fc7cb348b [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 14:47:31.025384 2794 tablet_replica.cc:330] T 86b00e18f13b4b9eb9b39010f3e6f17f P 5aa58728f3b74083b017cdcd573bc626: stopping tablet replica
I20260620 14:47:31.025631 2795 tablet_replica.cc:330] T 86b00e18f13b4b9eb9b39010f3e6f17f P e5717b124bc7489e8817304e3108a0eb: stopping tablet replica
I20260620 14:47:31.025629 2794 raft_consensus.cc:2229] T 86b00e18f13b4b9eb9b39010f3e6f17f P 5aa58728f3b74083b017cdcd573bc626 [term 1 LEADER]: Raft consensus shutting down.
I20260620 14:47:31.025852 2795 raft_consensus.cc:2229] T 86b00e18f13b4b9eb9b39010f3e6f17f P e5717b124bc7489e8817304e3108a0eb [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 14:47:31.026091 2795 raft_consensus.cc:2258] T 86b00e18f13b4b9eb9b39010f3e6f17f P e5717b124bc7489e8817304e3108a0eb [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 14:47:31.026289 2794 raft_consensus.cc:2258] T 86b00e18f13b4b9eb9b39010f3e6f17f P 5aa58728f3b74083b017cdcd573bc626 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 14:47:31.028671 2795 ts_tablet_manager.cc:1905] T 86b00e18f13b4b9eb9b39010f3e6f17f P e5717b124bc7489e8817304e3108a0eb: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 14:47:31.028721 2794 ts_tablet_manager.cc:1905] T 86b00e18f13b4b9eb9b39010f3e6f17f P 5aa58728f3b74083b017cdcd573bc626: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 14:47:31.028755 2796 ts_tablet_manager.cc:1905] T 86b00e18f13b4b9eb9b39010f3e6f17f P 7f1b44e3bb694105afe8822fc7cb348b: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 14:47:31.035712 2794 ts_tablet_manager.cc:1918] T 86b00e18f13b4b9eb9b39010f3e6f17f P 5aa58728f3b74083b017cdcd573bc626: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.8
I20260620 14:47:31.035760 2796 ts_tablet_manager.cc:1918] T 86b00e18f13b4b9eb9b39010f3e6f17f P 7f1b44e3bb694105afe8822fc7cb348b: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.8
I20260620 14:47:31.035912 2794 log.cc:1196] T 86b00e18f13b4b9eb9b39010f3e6f17f P 5aa58728f3b74083b017cdcd573bc626: Deleting WAL directory at /tmp/dist-test-taskUci3DU/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781966829091300-388-0/minicluster-data/ts-0/wal/wals/86b00e18f13b4b9eb9b39010f3e6f17f
I20260620 14:47:31.035913 2796 log.cc:1196] T 86b00e18f13b4b9eb9b39010f3e6f17f P 7f1b44e3bb694105afe8822fc7cb348b: Deleting WAL directory at /tmp/dist-test-taskUci3DU/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781966829091300-388-0/minicluster-data/ts-2/wal/wals/86b00e18f13b4b9eb9b39010f3e6f17f
I20260620 14:47:31.036334 2794 ts_tablet_manager.cc:1939] T 86b00e18f13b4b9eb9b39010f3e6f17f P 5aa58728f3b74083b017cdcd573bc626: Deleting consensus metadata
I20260620 14:47:31.036365 2796 ts_tablet_manager.cc:1939] T 86b00e18f13b4b9eb9b39010f3e6f17f P 7f1b44e3bb694105afe8822fc7cb348b: Deleting consensus metadata
I20260620 14:47:31.037145 2795 ts_tablet_manager.cc:1918] T 86b00e18f13b4b9eb9b39010f3e6f17f P e5717b124bc7489e8817304e3108a0eb: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.8
I20260620 14:47:31.037371 2795 log.cc:1196] T 86b00e18f13b4b9eb9b39010f3e6f17f P e5717b124bc7489e8817304e3108a0eb: Deleting WAL directory at /tmp/dist-test-taskUci3DU/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781966829091300-388-0/minicluster-data/ts-1/wal/wals/86b00e18f13b4b9eb9b39010f3e6f17f
I20260620 14:47:31.037539 2130 catalog_manager.cc:4841] TS 7f1b44e3bb694105afe8822fc7cb348b (127.0.97.3:32947): tablet 86b00e18f13b4b9eb9b39010f3e6f17f (table test-table [id=4e5193bde999478bae94c03557749add]) successfully deleted
I20260620 14:47:31.037798 2130 catalog_manager.cc:4841] TS 5aa58728f3b74083b017cdcd573bc626 (127.0.97.1:35927): tablet 86b00e18f13b4b9eb9b39010f3e6f17f (table test-table [id=4e5193bde999478bae94c03557749add]) successfully deleted
I20260620 14:47:31.037837 2795 ts_tablet_manager.cc:1939] T 86b00e18f13b4b9eb9b39010f3e6f17f P e5717b124bc7489e8817304e3108a0eb: Deleting consensus metadata
I20260620 14:47:31.038731 2129 catalog_manager.cc:4841] TS e5717b124bc7489e8817304e3108a0eb (127.0.97.2:37155): tablet 86b00e18f13b4b9eb9b39010f3e6f17f (table test-table [id=4e5193bde999478bae94c03557749add]) successfully deleted
Jun 20 14:47:31 dist-test-slave-q9j6 krb5kdc[2100](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781966849, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.0.97.1@KRBTEST.COM
I20260620 14:47:31.051453 2798 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:47:31.013928 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39640 (local address 127.0.97.1:35927)
0620 14:47:31.027335 (+ 13407us) server_negotiation.cc:206] Beginning negotiation
0620 14:47:31.027339 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 14:47:31.027864 (+ 525us) server_negotiation.cc:407] Connection header received
0620 14:47:31.028002 (+ 138us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:47:31.028006 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:47:31.028194 (+ 188us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:47:31.028428 (+ 234us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:47:31.030328 (+ 1900us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:47:31.031528 (+ 1200us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:47:31.033429 (+ 1901us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:47:31.034089 (+ 660us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:47:31.034209 (+ 120us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:47:31.045791 (+ 11582us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 14:47:31.045806 (+ 15us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 14:47:31.045823 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 14:47:31.045890 (+ 67us) server_negotiation.cc:922] calling sasl_server_start()
0620 14:47:31.049050 (+ 3160us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:47:31.049713 (+ 663us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:47:31.049721 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:47:31.049723 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:47:31.049889 (+ 166us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:47:31.050211 (+ 322us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:47:31.050214 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:47:31.050216 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:47:31.050571 (+ 355us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 14:47:31.050768 (+ 197us) server_negotiation.cc:1035] Waiting for connection context
0620 14:47:31.051027 (+ 259us) server_negotiation.cc:299] Negotiation successful
0620 14:47:31.051190 (+ 163us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":13291,"thread_start_us":13153,"threads_started":1}
W20260620 14:47:31.052143 2347 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:39640
W20260620 14:47:31.052795 388 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskUci3DU/build/asan/bin/kudu pid 2216: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260620 14:47:31.052999 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskUci3DU/build/asan/bin/kudu with pid 2216
I20260620 14:47:31.075574 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskUci3DU/build/asan/bin/kudu pid 2356
Jun 20 14:47:31 dist-test-slave-q9j6 krb5kdc[2100](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781966849, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.0.97.2@KRBTEST.COM
I20260620 14:47:31.091984 2799 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:47:31.076367 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58450 (local address 127.0.97.2:37155)
0620 14:47:31.076908 (+ 541us) server_negotiation.cc:206] Beginning negotiation
0620 14:47:31.076913 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 14:47:31.076930 (+ 17us) server_negotiation.cc:407] Connection header received
0620 14:47:31.076983 (+ 53us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:47:31.076987 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:47:31.077053 (+ 66us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:47:31.077146 (+ 93us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:47:31.078339 (+ 1193us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:47:31.079076 (+ 737us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:47:31.080436 (+ 1360us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:47:31.080919 (+ 483us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:47:31.081020 (+ 101us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:47:31.085470 (+ 4450us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 14:47:31.085484 (+ 14us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 14:47:31.085500 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 14:47:31.085538 (+ 38us) server_negotiation.cc:922] calling sasl_server_start()
0620 14:47:31.088864 (+ 3326us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:47:31.089755 (+ 891us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:47:31.089764 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:47:31.089768 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:47:31.089925 (+ 157us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:47:31.090303 (+ 378us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:47:31.090308 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:47:31.090312 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:47:31.090765 (+ 453us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 14:47:31.091055 (+ 290us) server_negotiation.cc:1035] Waiting for connection context
0620 14:47:31.091369 (+ 314us) server_negotiation.cc:299] Negotiation successful
0620 14:47:31.091590 (+ 221us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":432,"thread_start_us":306,"threads_started":1}
W20260620 14:47:31.092677 2486 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:58450
W20260620 14:47:31.093372 388 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskUci3DU/build/asan/bin/kudu pid 2356: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260620 14:47:31.093546 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskUci3DU/build/asan/bin/kudu with pid 2356
I20260620 14:47:31.115204 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskUci3DU/build/asan/bin/kudu pid 2495
Jun 20 14:47:31 dist-test-slave-q9j6 krb5kdc[2100](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781966849, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.0.97.3@KRBTEST.COM
I20260620 14:47:31.130745 2800 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:47:31.116084 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37622 (local address 127.0.97.3:32947)
0620 14:47:31.116543 (+ 459us) server_negotiation.cc:206] Beginning negotiation
0620 14:47:31.116547 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 14:47:31.116564 (+ 17us) server_negotiation.cc:407] Connection header received
0620 14:47:31.116617 (+ 53us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:47:31.116621 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:47:31.116684 (+ 63us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:47:31.116773 (+ 89us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:47:31.118028 (+ 1255us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:47:31.118739 (+ 711us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:47:31.120555 (+ 1816us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:47:31.120935 (+ 380us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:47:31.121028 (+ 93us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:47:31.125257 (+ 4229us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 14:47:31.125273 (+ 16us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 14:47:31.125289 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 14:47:31.125326 (+ 37us) server_negotiation.cc:922] calling sasl_server_start()
0620 14:47:31.128266 (+ 2940us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:47:31.128949 (+ 683us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:47:31.128956 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:47:31.128958 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:47:31.129084 (+ 126us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:47:31.129413 (+ 329us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:47:31.129417 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:47:31.129419 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:47:31.129753 (+ 334us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 14:47:31.129973 (+ 220us) server_negotiation.cc:1035] Waiting for connection context
0620 14:47:31.130243 (+ 270us) server_negotiation.cc:299] Negotiation successful
0620 14:47:31.130409 (+ 166us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":345,"thread_start_us":230,"threads_started":1}
W20260620 14:47:31.131428 2625 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:37622
W20260620 14:47:31.132136 388 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskUci3DU/build/asan/bin/kudu pid 2495: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260620 14:47:31.132295 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskUci3DU/build/asan/bin/kudu with pid 2495
I20260620 14:47:31.159386 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskUci3DU/build/asan/bin/kudu pid 2113
I20260620 14:47:31.160677 2185 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 14:47:31.282797 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskUci3DU/build/asan/bin/kudu with pid 2113
2026-06-20T14:47:31Z chronyd exiting
[ OK ] SecurityITest.TxnSmokeWithDifferentUserTypes (8210 ms)
[ RUN ] SecurityITest.TestNoKerberosCredentials
Loading random data
Initializing database '/tmp/dist-test-taskUci3DU/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781966829091300-388-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 20 14:47:31 dist-test-slave-q9j6 krb5kdc[2804](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 20 14:47:31 dist-test-slave-q9j6 krb5kdc[2804](info): set up 2 sockets
Jun 20 14:47:31 dist-test-slave-q9j6 krb5kdc[2804](info): commencing operation
krb5kdc: starting...
W20260620 14:47:32.451953 388 mini_kdc.cc:121] Time spent starting KDC: real 1.124s user 0.000s sys 0.019s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskUci3DU/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781966829091300-388-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskUci3DU/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781966829091300-388-0/krb5kdc/test-user.keytab.
Jun 20 14:47:32 dist-test-slave-q9j6 krb5kdc[2804](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781966852, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-20T14:47:32Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T14:47:32Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.97.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskUci3DU/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781966829091300-388-0/krb5kdc/kudu_127.0.97.62.keytab.
Entry for principal kudu/127.0.97.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskUci3DU/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781966829091300-388-0/krb5kdc/kudu_127.0.97.62.keytab.
I20260620 14:47:32.643664 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskUci3DU/build/asan/bin/kudu
/tmp/dist-test-taskUci3DU/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskUci3DU/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781966829091300-388-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskUci3DU/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781966829091300-388-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskUci3DU/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781966829091300-388-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskUci3DU/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781966829091300-388-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:36239
--webserver_interface=127.0.97.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskUci3DU/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781966829091300-388-0/krb5kdc/kudu_127.0.97.62.keytab
--principal=kudu/127.0.97.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:43699
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.97.62:36239
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskUci3DU/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781966829091300-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskUci3DU/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781966829091300-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskUci3DU/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781966829091300-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 14:47:32.974242 2817 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 14:47:32.974615 2817 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 14:47:32.974772 2817 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 14:47:32.982141 2817 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 14:47:32.982334 2817 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 14:47:32.982432 2817 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 14:47:32.982527 2817 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 14:47:32.982599 2817 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 14:47:32.992939 2817 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:43699
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskUci3DU/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781966829091300-388-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskUci3DU/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781966829091300-388-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.97.62:36239
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskUci3DU/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781966829091300-388-0/krb5kdc/kudu_127.0.97.62.keytab
--principal=kudu/127.0.97.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:36239
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskUci3DU/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781966829091300-388-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.97.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskUci3DU/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781966829091300-388-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 13:41:20 UTC on ccycloud.quasar-vdagoi.root.comops.site
build id 4457462
ASAN enabled
I20260620 14:47:32.994418 2817 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 14:47:32.995841 2817 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 14:47:33.007899 2822 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 14:47:33.008001 2825 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 14:47:33.007958 2823 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 14:47:33.008822 2817 server_base.cc:1034] running on GCE node
I20260620 14:47:33.009620 2817 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 14:47:33.011050 2817 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 14:47:33.012255 2817 hybrid_clock.cc:648] HybridClock initialized: now 1781966853012180 us; error 63 us; skew 500 ppm
I20260620 14:47:33.012601 2817 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 14:47:33 dist-test-slave-q9j6 krb5kdc[2804](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781966853, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 14:47:33.015763 2817 init.cc:377] Logged in from keytab as kudu/127.0.97.62@KRBTEST.COM (short username kudu)
I20260620 14:47:33.017197 2817 webserver.cc:460] Webserver started at http://127.0.97.62:41109/ using document root <none> and password file <none>
I20260620 14:47:33.017884 2817 fs_manager.cc:359] Metadata directory not provided
I20260620 14:47:33.017983 2817 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 14:47:33.018240 2817 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 14:47:33.021302 2817 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskUci3DU/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781966829091300-388-0/minicluster-data/master-0/data/instance:
uuid: "6834822aff534195a61c8fd66c53e02d"
format_stamp: "Formatted at 2026-06-20 14:47:33 on dist-test-slave-q9j6"
server_key: "be485bfb85b2af8b49d2b3040f143d75"
server_key_iv: "9b79bff67ac7174524f89dfa34af165a"
server_key_version: "encryptionkey@0"
I20260620 14:47:33.022076 2817 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskUci3DU/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781966829091300-388-0/minicluster-data/master-0/wal/instance:
uuid: "6834822aff534195a61c8fd66c53e02d"
format_stamp: "Formatted at 2026-06-20 14:47:33 on dist-test-slave-q9j6"
server_key: "be485bfb85b2af8b49d2b3040f143d75"
server_key_iv: "9b79bff67ac7174524f89dfa34af165a"
server_key_version: "encryptionkey@0"
I20260620 14:47:33.027436 2817 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.007s sys 0.000s
I20260620 14:47:33.031006 2832 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 14:47:33.032598 2817 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260620 14:47:33.032801 2817 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskUci3DU/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781966829091300-388-0/minicluster-data/master-0/data,/tmp/dist-test-taskUci3DU/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781966829091300-388-0/minicluster-data/master-0/wal
uuid: "6834822aff534195a61c8fd66c53e02d"
format_stamp: "Formatted at 2026-06-20 14:47:33 on dist-test-slave-q9j6"
server_key: "be485bfb85b2af8b49d2b3040f143d75"
server_key_iv: "9b79bff67ac7174524f89dfa34af165a"
server_key_version: "encryptionkey@0"
I20260620 14:47:33.033010 2817 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskUci3DU/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781966829091300-388-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskUci3DU/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781966829091300-388-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskUci3DU/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781966829091300-388-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 14:47:33.064520 2817 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 14:47:33.068823 2817 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 14:47:33.069075 2817 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 14:47:33.087567 2817 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.62:36239
I20260620 14:47:33.087589 2883 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.62:36239 every 8 connection(s)
I20260620 14:47:33.089148 2817 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskUci3DU/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781966829091300-388-0/minicluster-data/master-0/data/info.pb
I20260620 14:47:33.093047 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskUci3DU/build/asan/bin/kudu as pid 2817
I20260620 14:47:33.093266 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskUci3DU/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781966829091300-388-0/minicluster-data/master-0/wal/instance
I20260620 14:47:33.093456 2884 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 14:47:33.094066 388 external_mini_cluster.cc:1428] Setting key 946271d1af9885a163f8992e253e175f
I20260620 14:47:33.106669 2884 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 6834822aff534195a61c8fd66c53e02d: Bootstrap starting.
Jun 20 14:47:33 dist-test-slave-q9j6 krb5kdc[2804](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781966852, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260620 14:47:33.110265 2884 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 6834822aff534195a61c8fd66c53e02d: Neither blocks nor log segments found. Creating new log.
I20260620 14:47:33.111464 2884 log.cc:824] T 00000000000000000000000000000000 P 6834822aff534195a61c8fd66c53e02d: Log is configured to *not* fsync() on all Append() calls
I20260620 14:47:33.114609 2884 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 6834822aff534195a61c8fd66c53e02d: No bootstrap required, opened a new log
I20260620 14:47:33.116447 2887 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:47:33.096242 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46820 (local address 127.0.97.62:36239)
0620 14:47:33.097051 (+ 809us) server_negotiation.cc:206] Beginning negotiation
0620 14:47:33.097066 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0620 14:47:33.097113 (+ 47us) server_negotiation.cc:407] Connection header received
0620 14:47:33.098641 (+ 1528us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:47:33.098667 (+ 26us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:47:33.099128 (+ 461us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:47:33.099566 (+ 438us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:47:33.101094 (+ 1528us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:47:33.102404 (+ 1310us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:47:33.103709 (+ 1305us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:47:33.104480 (+ 771us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:47:33.104715 (+ 235us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:47:33.108757 (+ 4042us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 14:47:33.108766 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 14:47:33.108783 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 14:47:33.108833 (+ 50us) server_negotiation.cc:922] calling sasl_server_start()
0620 14:47:33.112076 (+ 3243us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:47:33.112859 (+ 783us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:47:33.112866 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:47:33.112875 (+ 9us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:47:33.113050 (+ 175us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:47:33.113460 (+ 410us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:47:33.113464 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:47:33.113466 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:47:33.114013 (+ 547us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 14:47:33.114342 (+ 329us) server_negotiation.cc:1035] Waiting for connection context
0620 14:47:33.114798 (+ 456us) server_negotiation.cc:299] Negotiation successful
0620 14:47:33.115321 (+ 523us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":605,"thread_start_us":360,"threads_started":1}
I20260620 14:47:33.122839 2884 raft_consensus.cc:348] T 00000000000000000000000000000000 P 6834822aff534195a61c8fd66c53e02d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6834822aff534195a61c8fd66c53e02d" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 36239 } }
I20260620 14:47:33.123414 2884 raft_consensus.cc:374] T 00000000000000000000000000000000 P 6834822aff534195a61c8fd66c53e02d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 14:47:33.123553 2884 raft_consensus.cc:729] T 00000000000000000000000000000000 P 6834822aff534195a61c8fd66c53e02d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6834822aff534195a61c8fd66c53e02d, State: Initialized, Role: FOLLOWER
I20260620 14:47:33.124213 2884 consensus_queue.cc:260] T 00000000000000000000000000000000 P 6834822aff534195a61c8fd66c53e02d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6834822aff534195a61c8fd66c53e02d" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 36239 } }
I20260620 14:47:33.124473 2884 raft_consensus.cc:388] T 00000000000000000000000000000000 P 6834822aff534195a61c8fd66c53e02d [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 14:47:33.124606 2884 raft_consensus.cc:482] T 00000000000000000000000000000000 P 6834822aff534195a61c8fd66c53e02d [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 14:47:33.124785 2884 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 6834822aff534195a61c8fd66c53e02d [term 0 FOLLOWER]: Advancing to term 1
I20260620 14:47:33.127014 2884 raft_consensus.cc:504] T 00000000000000000000000000000000 P 6834822aff534195a61c8fd66c53e02d [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6834822aff534195a61c8fd66c53e02d" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 36239 } }
I20260620 14:47:33.127558 2884 leader_election.cc:302] T 00000000000000000000000000000000 P 6834822aff534195a61c8fd66c53e02d [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 6834822aff534195a61c8fd66c53e02d; no voters:
I20260620 14:47:33.128269 2884 leader_election.cc:288] T 00000000000000000000000000000000 P 6834822aff534195a61c8fd66c53e02d [CANDIDATE]: Term 1 election: Requested vote from peers
I20260620 14:47:33.128474 2889 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 6834822aff534195a61c8fd66c53e02d [term 1 FOLLOWER]: Leader election won for term 1
I20260620 14:47:33.129186 2889 raft_consensus.cc:686] T 00000000000000000000000000000000 P 6834822aff534195a61c8fd66c53e02d [term 1 LEADER]: Becoming Leader. State: Replica: 6834822aff534195a61c8fd66c53e02d, State: Running, Role: LEADER
I20260620 14:47:33.129863 2889 consensus_queue.cc:237] T 00000000000000000000000000000000 P 6834822aff534195a61c8fd66c53e02d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6834822aff534195a61c8fd66c53e02d" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 36239 } }
I20260620 14:47:33.130544 2884 sys_catalog.cc:564] T 00000000000000000000000000000000 P 6834822aff534195a61c8fd66c53e02d [sys.catalog]: configured and running, proceeding with master startup.
I20260620 14:47:33.132804 2890 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6834822aff534195a61c8fd66c53e02d [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "6834822aff534195a61c8fd66c53e02d" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6834822aff534195a61c8fd66c53e02d" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 36239 } } }
I20260620 14:47:33.133139 2890 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6834822aff534195a61c8fd66c53e02d [sys.catalog]: This master's current role is: LEADER
I20260620 14:47:33.134038 2895 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 14:47:33.133863 2891 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6834822aff534195a61c8fd66c53e02d [sys.catalog]: SysCatalogTable state changed. Reason: New leader 6834822aff534195a61c8fd66c53e02d. Latest consensus state: current_term: 1 leader_uuid: "6834822aff534195a61c8fd66c53e02d" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6834822aff534195a61c8fd66c53e02d" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 36239 } } }
I20260620 14:47:33.134544 2891 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6834822aff534195a61c8fd66c53e02d [sys.catalog]: This master's current role is: LEADER
I20260620 14:47:33.140467 2895 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 14:47:33.150175 2895 catalog_manager.cc:1347] Generated new cluster ID: f2728887e1744a73b80e426d64a12dad
I20260620 14:47:33.150321 2895 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 14:47:33.169363 2895 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 14:47:33.170691 2895 catalog_manager.cc:1504] Loading token signing keys...
I20260620 14:47:33.185402 2895 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 6834822aff534195a61c8fd66c53e02d: Generated new TSK 0
I20260620 14:47:33.186524 2895 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.0.97.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskUci3DU/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781966829091300-388-0/krb5kdc/kudu_127.0.97.1.keytab.
Entry for principal kudu/127.0.97.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskUci3DU/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781966829091300-388-0/krb5kdc/kudu_127.0.97.1.keytab.
I20260620 14:47:33.233292 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskUci3DU/build/asan/bin/kudu
/tmp/dist-test-taskUci3DU/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskUci3DU/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781966829091300-388-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskUci3DU/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781966829091300-388-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskUci3DU/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781966829091300-388-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskUci3DU/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781966829091300-388-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.1:0
--local_ip_for_outbound_sockets=127.0.97.1
--webserver_interface=127.0.97.1
--webserver_port=0
--tserver_master_addrs=127.0.97.62:36239
--keytab_file=/tmp/dist-test-taskUci3DU/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781966829091300-388-0/krb5kdc/kudu_127.0.97.1.keytab
--principal=kudu/127.0.97.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:43699
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskUci3DU/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781966829091300-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskUci3DU/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781966829091300-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskUci3DU/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781966829091300-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 14:47:33.573870 2910 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 14:47:33.574167 2910 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 14:47:33.574339 2910 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 14:47:33.581713 2910 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 14:47:33.581832 2910 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 14:47:33.581987 2910 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.1
I20260620 14:47:33.592196 2910 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:43699
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskUci3DU/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781966829091300-388-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskUci3DU/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781966829091300-388-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskUci3DU/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781966829091300-388-0/krb5kdc/kudu_127.0.97.1.keytab
--principal=kudu/127.0.97.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskUci3DU/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781966829091300-388-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.97.1
--webserver_port=0
--tserver_master_addrs=127.0.97.62:36239
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.1
--log_dir=/tmp/dist-test-taskUci3DU/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781966829091300-388-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 13:41:20 UTC on ccycloud.quasar-vdagoi.root.comops.site
build id 4457462
ASAN enabled
I20260620 14:47:33.593520 2910 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 14:47:33.594813 2910 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 14:47:33.608108 2916 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 14:47:33.608194 2918 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 14:47:33.608150 2915 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 14:47:33.609136 2910 server_base.cc:1034] running on GCE node
I20260620 14:47:33.609760 2910 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 14:47:33.610728 2910 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 14:47:33.611956 2910 hybrid_clock.cc:648] HybridClock initialized: now 1781966853611907 us; error 55 us; skew 500 ppm
I20260620 14:47:33.612284 2910 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 14:47:33 dist-test-slave-q9j6 krb5kdc[2804](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781966853, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 14:47:33.615473 2910 init.cc:377] Logged in from keytab as kudu/127.0.97.1@KRBTEST.COM (short username kudu)
I20260620 14:47:33.617054 2910 webserver.cc:460] Webserver started at http://127.0.97.1:42577/ using document root <none> and password file <none>
I20260620 14:47:33.617769 2910 fs_manager.cc:359] Metadata directory not provided
I20260620 14:47:33.617938 2910 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 14:47:33.618247 2910 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 14:47:33.621296 2910 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskUci3DU/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781966829091300-388-0/minicluster-data/ts-0/data/instance:
uuid: "ce3557902fe144feac450af8143b83c9"
format_stamp: "Formatted at 2026-06-20 14:47:33 on dist-test-slave-q9j6"
server_key: "71a69ccda0b74803826da4a5fe8d23bb"
server_key_iv: "0cc5243c154b616ad82bb7a95ca18bc7"
server_key_version: "encryptionkey@0"
I20260620 14:47:33.622131 2910 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskUci3DU/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781966829091300-388-0/minicluster-data/ts-0/wal/instance:
uuid: "ce3557902fe144feac450af8143b83c9"
format_stamp: "Formatted at 2026-06-20 14:47:33 on dist-test-slave-q9j6"
server_key: "71a69ccda0b74803826da4a5fe8d23bb"
server_key_iv: "0cc5243c154b616ad82bb7a95ca18bc7"
server_key_version: "encryptionkey@0"
I20260620 14:47:33.627748 2910 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.004s sys 0.004s
I20260620 14:47:33.631414 2925 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 14:47:33.633005 2910 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260620 14:47:33.633203 2910 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskUci3DU/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781966829091300-388-0/minicluster-data/ts-0/data,/tmp/dist-test-taskUci3DU/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781966829091300-388-0/minicluster-data/ts-0/wal
uuid: "ce3557902fe144feac450af8143b83c9"
format_stamp: "Formatted at 2026-06-20 14:47:33 on dist-test-slave-q9j6"
server_key: "71a69ccda0b74803826da4a5fe8d23bb"
server_key_iv: "0cc5243c154b616ad82bb7a95ca18bc7"
server_key_version: "encryptionkey@0"
I20260620 14:47:33.633424 2910 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskUci3DU/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781966829091300-388-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskUci3DU/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781966829091300-388-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskUci3DU/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781966829091300-388-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 14:47:33.661660 2910 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 14:47:33.665839 2910 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 14:47:33.666134 2910 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 14:47:33.667125 2910 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 14:47:33.668532 2910 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 14:47:33.668629 2910 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 14:47:33.668764 2910 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 14:47:33.668839 2910 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 14:47:33.701776 2910 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.1:37963
I20260620 14:47:33.701834 3037 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.1:37963 every 8 connection(s)
I20260620 14:47:33.703159 2910 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskUci3DU/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781966829091300-388-0/minicluster-data/ts-0/data/info.pb
I20260620 14:47:33.713395 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskUci3DU/build/asan/bin/kudu as pid 2910
I20260620 14:47:33.713595 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskUci3DU/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781966829091300-388-0/minicluster-data/ts-0/wal/instance
I20260620 14:47:33.714250 388 external_mini_cluster.cc:1428] Setting key 5b8cb6e78a9d6229a8478e8fd4a70991
Jun 20 14:47:33 dist-test-slave-q9j6 krb5kdc[2804](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781966853, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.1@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260620 14:47:33.722819 3040 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:47:33.705782 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.1:52621 (local address 127.0.97.62:36239)
0620 14:47:33.706238 (+ 456us) server_negotiation.cc:206] Beginning negotiation
0620 14:47:33.706243 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 14:47:33.707154 (+ 911us) server_negotiation.cc:407] Connection header received
0620 14:47:33.709047 (+ 1893us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:47:33.709053 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:47:33.709144 (+ 91us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:47:33.709259 (+ 115us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:47:33.710746 (+ 1487us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:47:33.711447 (+ 701us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:47:33.713088 (+ 1641us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:47:33.713607 (+ 519us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:47:33.713746 (+ 139us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:47:33.717232 (+ 3486us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 14:47:33.717237 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 14:47:33.717240 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 14:47:33.717270 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0620 14:47:33.719792 (+ 2522us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:47:33.720510 (+ 718us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:47:33.720521 (+ 11us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:47:33.720523 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:47:33.720631 (+ 108us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:47:33.721063 (+ 432us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:47:33.721068 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:47:33.721071 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:47:33.721368 (+ 297us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 14:47:33.721539 (+ 171us) server_negotiation.cc:1035] Waiting for connection context
0620 14:47:33.722242 (+ 703us) server_negotiation.cc:299] Negotiation successful
0620 14:47:33.722466 (+ 224us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":353,"thread_start_us":244,"threads_started":1}
I20260620 14:47:33.723903 3041 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:47:33.706279 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:36239 (local address 127.0.97.1:52621)
0620 14:47:33.706994 (+ 715us) negotiation.cc:107] Waiting for socket to connect
0620 14:47:33.707035 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0620 14:47:33.708761 (+ 1726us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 14:47:33.709495 (+ 734us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 14:47:33.709503 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 14:47:33.710030 (+ 527us) client_negotiation.cc:189] Negotiated authn=SASL
0620 14:47:33.710579 (+ 549us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:47:33.710596 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:47:33.711649 (+ 1053us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:47:33.711653 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:47:33.712903 (+ 1250us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:47:33.712919 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:47:33.713871 (+ 952us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:47:33.713879 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:47:33.714117 (+ 238us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:47:33.714849 (+ 732us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 14:47:33.714878 (+ 29us) client_negotiation.cc:656] calling sasl_client_start()
0620 14:47:33.717016 (+ 2138us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 14:47:33.719932 (+ 2916us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 14:47:33.719941 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 14:47:33.719961 (+ 20us) client_negotiation.cc:760] calling sasl_client_step()
0620 14:47:33.720388 (+ 427us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 14:47:33.720737 (+ 349us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 14:47:33.720742 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 14:47:33.720746 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0620 14:47:33.720949 (+ 203us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 14:47:33.721493 (+ 544us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 14:47:33.721501 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 14:47:33.721853 (+ 352us) client_negotiation.cc:769] Sending connection context
0620 14:47:33.722120 (+ 267us) client_negotiation.cc:240] Negotiation successful
0620 14:47:33.722677 (+ 557us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":482,"thread_start_us":271,"threads_started":1}
I20260620 14:47:33.725515 3038 heartbeater.cc:347] Connected to a master server at 127.0.97.62:36239
I20260620 14:47:33.725826 3038 heartbeater.cc:464] Registering TS with master...
I20260620 14:47:33.726544 3038 heartbeater.cc:511] Master 127.0.97.62:36239 requested a full tablet report, sending...
I20260620 14:47:33.729046 2849 ts_manager.cc:194] Registered new tserver with Master: ce3557902fe144feac450af8143b83c9 (127.0.97.1:37963)
I20260620 14:47:33.731295 2849 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.97.1@KRBTEST.COM'} at 127.0.97.1:52621
WARNING: no policy specified for kudu/127.0.97.2@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskUci3DU/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781966829091300-388-0/krb5kdc/kudu_127.0.97.2.keytab.
Entry for principal kudu/127.0.97.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskUci3DU/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781966829091300-388-0/krb5kdc/kudu_127.0.97.2.keytab.
I20260620 14:47:33.756835 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskUci3DU/build/asan/bin/kudu
/tmp/dist-test-taskUci3DU/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskUci3DU/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781966829091300-388-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskUci3DU/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781966829091300-388-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskUci3DU/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781966829091300-388-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskUci3DU/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781966829091300-388-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.2:0
--local_ip_for_outbound_sockets=127.0.97.2
--webserver_interface=127.0.97.2
--webserver_port=0
--tserver_master_addrs=127.0.97.62:36239
--keytab_file=/tmp/dist-test-taskUci3DU/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781966829091300-388-0/krb5kdc/kudu_127.0.97.2.keytab
--principal=kudu/127.0.97.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:43699
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskUci3DU/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781966829091300-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskUci3DU/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781966829091300-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskUci3DU/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781966829091300-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 14:47:34.086970 3044 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 14:47:34.087263 3044 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 14:47:34.087428 3044 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 14:47:34.095122 3044 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 14:47:34.095250 3044 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 14:47:34.095422 3044 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.2
I20260620 14:47:34.105671 3044 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:43699
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskUci3DU/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781966829091300-388-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskUci3DU/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781966829091300-388-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskUci3DU/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781966829091300-388-0/krb5kdc/kudu_127.0.97.2.keytab
--principal=kudu/127.0.97.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskUci3DU/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781966829091300-388-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.97.2
--webserver_port=0
--tserver_master_addrs=127.0.97.62:36239
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.2
--log_dir=/tmp/dist-test-taskUci3DU/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781966829091300-388-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 13:41:20 UTC on ccycloud.quasar-vdagoi.root.comops.site
build id 4457462
ASAN enabled
I20260620 14:47:34.107028 3044 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 14:47:34.108433 3044 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 14:47:34.122073 3049 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 14:47:34.122115 3050 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 14:47:34.122181 3052 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 14:47:34.123174 3044 server_base.cc:1034] running on GCE node
I20260620 14:47:34.123878 3044 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 14:47:34.124879 3044 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 14:47:34.126114 3044 hybrid_clock.cc:648] HybridClock initialized: now 1781966854126062 us; error 51 us; skew 500 ppm
I20260620 14:47:34.126446 3044 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 14:47:34 dist-test-slave-q9j6 krb5kdc[2804](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781966854, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 14:47:34.130151 3044 init.cc:377] Logged in from keytab as kudu/127.0.97.2@KRBTEST.COM (short username kudu)
I20260620 14:47:34.131798 3044 webserver.cc:460] Webserver started at http://127.0.97.2:45771/ using document root <none> and password file <none>
I20260620 14:47:34.132517 3044 fs_manager.cc:359] Metadata directory not provided
I20260620 14:47:34.132647 3044 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 14:47:34.132968 3044 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 14:47:34.136175 3044 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskUci3DU/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781966829091300-388-0/minicluster-data/ts-1/data/instance:
uuid: "54ac5fee892944babc99e7430d7f22d3"
format_stamp: "Formatted at 2026-06-20 14:47:34 on dist-test-slave-q9j6"
server_key: "5ff4ffa8cb35579a6ebcb9b2bc3076cb"
server_key_iv: "7f1618fc039b7ec31c148eb9ae79715f"
server_key_version: "encryptionkey@0"
I20260620 14:47:34.137140 3044 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskUci3DU/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781966829091300-388-0/minicluster-data/ts-1/wal/instance:
uuid: "54ac5fee892944babc99e7430d7f22d3"
format_stamp: "Formatted at 2026-06-20 14:47:34 on dist-test-slave-q9j6"
server_key: "5ff4ffa8cb35579a6ebcb9b2bc3076cb"
server_key_iv: "7f1618fc039b7ec31c148eb9ae79715f"
server_key_version: "encryptionkey@0"
I20260620 14:47:34.142741 3044 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.001s
I20260620 14:47:34.146374 3059 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 14:47:34.147908 3044 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260620 14:47:34.148084 3044 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskUci3DU/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781966829091300-388-0/minicluster-data/ts-1/data,/tmp/dist-test-taskUci3DU/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781966829091300-388-0/minicluster-data/ts-1/wal
uuid: "54ac5fee892944babc99e7430d7f22d3"
format_stamp: "Formatted at 2026-06-20 14:47:34 on dist-test-slave-q9j6"
server_key: "5ff4ffa8cb35579a6ebcb9b2bc3076cb"
server_key_iv: "7f1618fc039b7ec31c148eb9ae79715f"
server_key_version: "encryptionkey@0"
I20260620 14:47:34.148272 3044 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskUci3DU/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781966829091300-388-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskUci3DU/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781966829091300-388-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskUci3DU/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781966829091300-388-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 14:47:34.181360 3044 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 14:47:34.185933 3044 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 14:47:34.186214 3044 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 14:47:34.187222 3044 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 14:47:34.188624 3044 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 14:47:34.188694 3044 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 14:47:34.188823 3044 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 14:47:34.188879 3044 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 14:47:34.220009 3044 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.2:37883
I20260620 14:47:34.220044 3171 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.2:37883 every 8 connection(s)
I20260620 14:47:34.221320 3044 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskUci3DU/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781966829091300-388-0/minicluster-data/ts-1/data/info.pb
I20260620 14:47:34.225804 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskUci3DU/build/asan/bin/kudu as pid 3044
I20260620 14:47:34.225972 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskUci3DU/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781966829091300-388-0/minicluster-data/ts-1/wal/instance
I20260620 14:47:34.226399 388 external_mini_cluster.cc:1428] Setting key 75ded582e11f7db044969398961a5ce1
Jun 20 14:47:34 dist-test-slave-q9j6 krb5kdc[2804](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781966854, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.2@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260620 14:47:34.242369 3174 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:47:34.223913 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.2:53721 (local address 127.0.97.62:36239)
0620 14:47:34.224340 (+ 427us) server_negotiation.cc:206] Beginning negotiation
0620 14:47:34.224345 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 14:47:34.225358 (+ 1013us) server_negotiation.cc:407] Connection header received
0620 14:47:34.227444 (+ 2086us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:47:34.227450 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:47:34.227542 (+ 92us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:47:34.227692 (+ 150us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:47:34.229351 (+ 1659us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:47:34.230066 (+ 715us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:47:34.231896 (+ 1830us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:47:34.232327 (+ 431us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:47:34.232465 (+ 138us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:47:34.235972 (+ 3507us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 14:47:34.235977 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 14:47:34.235981 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 14:47:34.236014 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0620 14:47:34.239181 (+ 3167us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:47:34.240019 (+ 838us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:47:34.240025 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:47:34.240029 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:47:34.240168 (+ 139us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:47:34.240597 (+ 429us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:47:34.240602 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:47:34.240606 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:47:34.240870 (+ 264us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 14:47:34.241027 (+ 157us) server_negotiation.cc:1035] Waiting for connection context
0620 14:47:34.241736 (+ 709us) server_negotiation.cc:299] Negotiation successful
0620 14:47:34.241963 (+ 227us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":1,"server-negotiator.queue_time_us":323,"thread_start_us":220,"threads_started":1}
WARNING: no policy specified for kudu/127.0.97.3@KRBTEST.COM; defaulting to no policy
I20260620 14:47:34.243381 3175 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:47:34.224435 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:36239 (local address 127.0.97.2:53721)
0620 14:47:34.225198 (+ 763us) negotiation.cc:107] Waiting for socket to connect
0620 14:47:34.225236 (+ 38us) client_negotiation.cc:174] Beginning negotiation
0620 14:47:34.227172 (+ 1936us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 14:47:34.227940 (+ 768us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 14:47:34.227949 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 14:47:34.228601 (+ 652us) client_negotiation.cc:189] Negotiated authn=SASL
0620 14:47:34.229182 (+ 581us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:47:34.229200 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:47:34.230228 (+ 1028us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:47:34.230232 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:47:34.231747 (+ 1515us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:47:34.231761 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:47:34.232570 (+ 809us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:47:34.232576 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:47:34.232805 (+ 229us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:47:34.233548 (+ 743us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 14:47:34.233575 (+ 27us) client_negotiation.cc:656] calling sasl_client_start()
0620 14:47:34.235735 (+ 2160us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 14:47:34.239331 (+ 3596us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 14:47:34.239342 (+ 11us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 14:47:34.239362 (+ 20us) client_negotiation.cc:760] calling sasl_client_step()
0620 14:47:34.239886 (+ 524us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 14:47:34.240278 (+ 392us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 14:47:34.240283 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 14:47:34.240287 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0620 14:47:34.240481 (+ 194us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 14:47:34.241000 (+ 519us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 14:47:34.241008 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 14:47:34.241368 (+ 360us) client_negotiation.cc:769] Sending connection context
0620 14:47:34.241655 (+ 287us) client_negotiation.cc:240] Negotiation successful
0620 14:47:34.242203 (+ 548us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":522,"thread_start_us":318,"threads_started":1}
I20260620 14:47:34.244886 3172 heartbeater.cc:347] Connected to a master server at 127.0.97.62:36239
I20260620 14:47:34.245215 3172 heartbeater.cc:464] Registering TS with master...
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.3@KRBTEST.COM" created.
I20260620 14:47:34.246106 3172 heartbeater.cc:511] Master 127.0.97.62:36239 requested a full tablet report, sending...
I20260620 14:47:34.247538 2849 ts_manager.cc:194] Registered new tserver with Master: 54ac5fee892944babc99e7430d7f22d3 (127.0.97.2:37883)
I20260620 14:47:34.248386 2849 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.97.2@KRBTEST.COM'} at 127.0.97.2:53721
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskUci3DU/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781966829091300-388-0/krb5kdc/kudu_127.0.97.3.keytab.
Entry for principal kudu/127.0.97.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskUci3DU/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781966829091300-388-0/krb5kdc/kudu_127.0.97.3.keytab.
I20260620 14:47:34.268151 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskUci3DU/build/asan/bin/kudu
/tmp/dist-test-taskUci3DU/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskUci3DU/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781966829091300-388-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskUci3DU/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781966829091300-388-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskUci3DU/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781966829091300-388-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskUci3DU/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781966829091300-388-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.3:0
--local_ip_for_outbound_sockets=127.0.97.3
--webserver_interface=127.0.97.3
--webserver_port=0
--tserver_master_addrs=127.0.97.62:36239
--keytab_file=/tmp/dist-test-taskUci3DU/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781966829091300-388-0/krb5kdc/kudu_127.0.97.3.keytab
--principal=kudu/127.0.97.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:43699
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskUci3DU/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781966829091300-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskUci3DU/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781966829091300-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskUci3DU/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781966829091300-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 14:47:34.597092 3178 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 14:47:34.597404 3178 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 14:47:34.597577 3178 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 14:47:34.605188 3178 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 14:47:34.605373 3178 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 14:47:34.605544 3178 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.3
I20260620 14:47:34.618510 3178 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:43699
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskUci3DU/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781966829091300-388-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskUci3DU/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781966829091300-388-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskUci3DU/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781966829091300-388-0/krb5kdc/kudu_127.0.97.3.keytab
--principal=kudu/127.0.97.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskUci3DU/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781966829091300-388-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.97.3
--webserver_port=0
--tserver_master_addrs=127.0.97.62:36239
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.3
--log_dir=/tmp/dist-test-taskUci3DU/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781966829091300-388-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 13:41:20 UTC on ccycloud.quasar-vdagoi.root.comops.site
build id 4457462
ASAN enabled
I20260620 14:47:34.620434 3178 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 14:47:34.622393 3178 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 14:47:34.636368 3183 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 14:47:34.636427 3186 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 14:47:34.636448 3184 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 14:47:34.637646 3178 server_base.cc:1034] running on GCE node
I20260620 14:47:34.638361 3178 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 14:47:34.639359 3178 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 14:47:34.640558 3178 hybrid_clock.cc:648] HybridClock initialized: now 1781966854640500 us; error 43 us; skew 500 ppm
I20260620 14:47:34.640887 3178 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 14:47:34 dist-test-slave-q9j6 krb5kdc[2804](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781966854, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 14:47:34.644001 3178 init.cc:377] Logged in from keytab as kudu/127.0.97.3@KRBTEST.COM (short username kudu)
I20260620 14:47:34.645416 3178 webserver.cc:460] Webserver started at http://127.0.97.3:39577/ using document root <none> and password file <none>
I20260620 14:47:34.646104 3178 fs_manager.cc:359] Metadata directory not provided
I20260620 14:47:34.646205 3178 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 14:47:34.646477 3178 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 14:47:34.649434 3178 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskUci3DU/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781966829091300-388-0/minicluster-data/ts-2/data/instance:
uuid: "8a3326767bc04c4b87d6ddf0955f246a"
format_stamp: "Formatted at 2026-06-20 14:47:34 on dist-test-slave-q9j6"
server_key: "2dd46e01281657b1e5eb41b8642f6f4a"
server_key_iv: "30f064afe85b3fb4e65976e2a9062e00"
server_key_version: "encryptionkey@0"
I20260620 14:47:34.650170 3178 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskUci3DU/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781966829091300-388-0/minicluster-data/ts-2/wal/instance:
uuid: "8a3326767bc04c4b87d6ddf0955f246a"
format_stamp: "Formatted at 2026-06-20 14:47:34 on dist-test-slave-q9j6"
server_key: "2dd46e01281657b1e5eb41b8642f6f4a"
server_key_iv: "30f064afe85b3fb4e65976e2a9062e00"
server_key_version: "encryptionkey@0"
I20260620 14:47:34.655475 3178 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.007s sys 0.000s
I20260620 14:47:34.658993 3193 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 14:47:34.660501 3178 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260620 14:47:34.660687 3178 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskUci3DU/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781966829091300-388-0/minicluster-data/ts-2/data,/tmp/dist-test-taskUci3DU/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781966829091300-388-0/minicluster-data/ts-2/wal
uuid: "8a3326767bc04c4b87d6ddf0955f246a"
format_stamp: "Formatted at 2026-06-20 14:47:34 on dist-test-slave-q9j6"
server_key: "2dd46e01281657b1e5eb41b8642f6f4a"
server_key_iv: "30f064afe85b3fb4e65976e2a9062e00"
server_key_version: "encryptionkey@0"
I20260620 14:47:34.660893 3178 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskUci3DU/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781966829091300-388-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskUci3DU/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781966829091300-388-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskUci3DU/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781966829091300-388-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 14:47:34.682890 3178 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 14:47:34.687057 3178 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 14:47:34.687376 3178 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 14:47:34.688434 3178 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 14:47:34.689872 3178 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 14:47:34.689955 3178 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 14:47:34.690121 3178 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 14:47:34.690171 3178 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 14:47:34.723093 3178 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.3:34829
I20260620 14:47:34.723111 3305 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.3:34829 every 8 connection(s)
I20260620 14:47:34.724490 3178 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskUci3DU/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781966829091300-388-0/minicluster-data/ts-2/data/info.pb
I20260620 14:47:34.726482 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskUci3DU/build/asan/bin/kudu as pid 3178
I20260620 14:47:34.726655 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskUci3DU/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781966829091300-388-0/minicluster-data/ts-2/wal/instance
I20260620 14:47:34.727224 388 external_mini_cluster.cc:1428] Setting key 07fe442b023c7d9bcfc16b924e054560
I20260620 14:47:34.734603 3038 heartbeater.cc:503] Master 127.0.97.62:36239 was elected leader, sending a full tablet report...
Jun 20 14:47:34 dist-test-slave-q9j6 krb5kdc[2804](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781966854, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.3@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260620 14:47:34.744954 3174 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:47:34.727314 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.3:38865 (local address 127.0.97.62:36239)
0620 14:47:34.727501 (+ 187us) server_negotiation.cc:206] Beginning negotiation
0620 14:47:34.727506 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 14:47:34.729053 (+ 1547us) server_negotiation.cc:407] Connection header received
0620 14:47:34.731138 (+ 2085us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:47:34.731142 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:47:34.731207 (+ 65us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:47:34.731289 (+ 82us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:47:34.732786 (+ 1497us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:47:34.733484 (+ 698us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:47:34.735029 (+ 1545us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:47:34.735544 (+ 515us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:47:34.735720 (+ 176us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:47:34.739395 (+ 3675us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 14:47:34.739400 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 14:47:34.739403 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 14:47:34.739433 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0620 14:47:34.742412 (+ 2979us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:47:34.743044 (+ 632us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:47:34.743048 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:47:34.743050 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:47:34.743160 (+ 110us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:47:34.743498 (+ 338us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:47:34.743501 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:47:34.743502 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:47:34.743825 (+ 323us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 14:47:34.743980 (+ 155us) server_negotiation.cc:1035] Waiting for connection context
0620 14:47:34.744554 (+ 574us) server_negotiation.cc:299] Negotiation successful
0620 14:47:34.744724 (+ 170us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":43}
I20260620 14:47:34.745940 3308 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:47:34.727981 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:36239 (local address 127.0.97.3:38865)
0620 14:47:34.728853 (+ 872us) negotiation.cc:107] Waiting for socket to connect
0620 14:47:34.728911 (+ 58us) client_negotiation.cc:174] Beginning negotiation
0620 14:47:34.730894 (+ 1983us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 14:47:34.731500 (+ 606us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 14:47:34.731509 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 14:47:34.732068 (+ 559us) client_negotiation.cc:189] Negotiated authn=SASL
0620 14:47:34.732630 (+ 562us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:47:34.732647 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:47:34.733600 (+ 953us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:47:34.733604 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:47:34.734872 (+ 1268us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:47:34.734902 (+ 30us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:47:34.735804 (+ 902us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:47:34.735812 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:47:34.736108 (+ 296us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:47:34.736891 (+ 783us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 14:47:34.736913 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0620 14:47:34.739190 (+ 2277us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 14:47:34.742530 (+ 3340us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 14:47:34.742537 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 14:47:34.742551 (+ 14us) client_negotiation.cc:760] calling sasl_client_step()
0620 14:47:34.742945 (+ 394us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 14:47:34.743241 (+ 296us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 14:47:34.743244 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 14:47:34.743246 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 14:47:34.743415 (+ 169us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 14:47:34.743933 (+ 518us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 14:47:34.743938 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 14:47:34.744238 (+ 300us) client_negotiation.cc:769] Sending connection context
0620 14:47:34.744472 (+ 234us) client_negotiation.cc:240] Negotiation successful
0620 14:47:34.744837 (+ 365us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":565,"thread_start_us":331,"threads_started":1}
I20260620 14:47:34.747115 3306 heartbeater.cc:347] Connected to a master server at 127.0.97.62:36239
I20260620 14:47:34.747406 3306 heartbeater.cc:464] Registering TS with master...
I20260620 14:47:34.748121 3306 heartbeater.cc:511] Master 127.0.97.62:36239 requested a full tablet report, sending...
I20260620 14:47:34.749378 2849 ts_manager.cc:194] Registered new tserver with Master: 8a3326767bc04c4b87d6ddf0955f246a (127.0.97.3:34829)
I20260620 14:47:34.750140 2849 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.97.3@KRBTEST.COM'} at 127.0.97.3:38865
I20260620 14:47:34.754384 388 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
W20260620 14:47:34.783869 3174 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 14:47:34.781486 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46832 (local address 127.0.97.62:36239)
0620 14:47:34.781665 (+ 179us) server_negotiation.cc:206] Beginning negotiation
0620 14:47:34.781670 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 14:47:34.782148 (+ 478us) server_negotiation.cc:407] Connection header received
0620 14:47:34.782312 (+ 164us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:47:34.782316 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:47:34.782386 (+ 70us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:47:34.782482 (+ 96us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:47:34.783666 (+ 1184us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:46832: BlockingRecv error: recv got EOF from 127.0.0.1:46832 (error 108)
Metrics: {"server-negotiator.queue_time_us":61}
I20260620 14:47:34.786966 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskUci3DU/build/asan/bin/kudu pid 2910
W20260620 14:47:34.789629 388 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskUci3DU/build/asan/bin/kudu pid 2910: Not authorized: Client connection negotiation failed: client connection to 127.0.97.1:37963: server requires authentication, but client does not have Kerberos credentials available
I20260620 14:47:34.789799 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskUci3DU/build/asan/bin/kudu with pid 2910
I20260620 14:47:34.805425 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskUci3DU/build/asan/bin/kudu pid 3044
W20260620 14:47:34.808238 388 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskUci3DU/build/asan/bin/kudu pid 3044: Not authorized: Client connection negotiation failed: client connection to 127.0.97.2:37883: server requires authentication, but client does not have Kerberos credentials available
I20260620 14:47:34.808404 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskUci3DU/build/asan/bin/kudu with pid 3044
I20260620 14:47:34.824576 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskUci3DU/build/asan/bin/kudu pid 3178
W20260620 14:47:34.827943 388 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskUci3DU/build/asan/bin/kudu pid 3178: Not authorized: Client connection negotiation failed: client connection to 127.0.97.3:34829: server requires authentication, but client does not have Kerberos credentials available
I20260620 14:47:34.828135 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskUci3DU/build/asan/bin/kudu with pid 3178
I20260620 14:47:34.841398 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskUci3DU/build/asan/bin/kudu pid 2817
I20260620 14:47:34.842686 2879 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 14:47:34.946761 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskUci3DU/build/asan/bin/kudu with pid 2817
2026-06-20T14:47:34Z chronyd exiting
[ OK ] SecurityITest.TestNoKerberosCredentials (3662 ms)
[ RUN ] SecurityITest.SaslPlainFallback
Loading random data
Initializing database '/tmp/dist-test-taskUci3DU/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1781966829091300-388-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 20 14:47:35 dist-test-slave-q9j6 krb5kdc[3323](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 20 14:47:35 dist-test-slave-q9j6 krb5kdc[3323](info): set up 2 sockets
Jun 20 14:47:35 dist-test-slave-q9j6 krb5kdc[3323](info): commencing operation
krb5kdc: starting...
W20260620 14:47:36.056119 388 mini_kdc.cc:121] Time spent starting KDC: real 1.066s user 0.003s sys 0.014s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskUci3DU/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1781966829091300-388-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskUci3DU/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1781966829091300-388-0/krb5kdc/test-user.keytab.
Jun 20 14:47:36 dist-test-slave-q9j6 krb5kdc[3323](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781966856, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-20T14:47:36Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T14:47:36Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.97.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskUci3DU/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1781966829091300-388-0/krb5kdc/kudu_127.0.97.62.keytab.
Entry for principal kudu/127.0.97.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskUci3DU/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1781966829091300-388-0/krb5kdc/kudu_127.0.97.62.keytab.
I20260620 14:47:36.253298 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskUci3DU/build/asan/bin/kudu
/tmp/dist-test-taskUci3DU/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskUci3DU/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1781966829091300-388-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskUci3DU/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1781966829091300-388-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskUci3DU/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1781966829091300-388-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskUci3DU/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1781966829091300-388-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:44827
--webserver_interface=127.0.97.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskUci3DU/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1781966829091300-388-0/krb5kdc/kudu_127.0.97.62.keytab
--principal=kudu/127.0.97.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:45529
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.97.62:44827
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc-authentication=optional
--user-acl=* with env {KRB5CCNAME=/tmp/dist-test-taskUci3DU/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1781966829091300-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskUci3DU/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1781966829091300-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskUci3DU/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1781966829091300-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 14:47:36.586652 3336 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 14:47:36.586962 3336 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 14:47:36.587160 3336 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 14:47:36.594643 3336 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 14:47:36.594770 3336 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 14:47:36.594866 3336 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 14:47:36.594980 3336 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 14:47:36.595085 3336 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 14:47:36.606166 3336 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:45529
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskUci3DU/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1781966829091300-388-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskUci3DU/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1781966829091300-388-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.97.62:44827
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskUci3DU/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1781966829091300-388-0/krb5kdc/kudu_127.0.97.62.keytab
--principal=kudu/127.0.97.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:44827
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskUci3DU/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1781966829091300-388-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.97.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskUci3DU/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1781966829091300-388-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 13:41:20 UTC on ccycloud.quasar-vdagoi.root.comops.site
build id 4457462
ASAN enabled
I20260620 14:47:36.607576 3336 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 14:47:36.609033 3336 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 14:47:36.624133 3342 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 14:47:36.624172 3344 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 14:47:36.625226 3336 server_base.cc:1034] running on GCE node
W20260620 14:47:36.625393 3341 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 14:47:36.626286 3336 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 14:47:36.627821 3336 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 14:47:36.629029 3336 hybrid_clock.cc:648] HybridClock initialized: now 1781966856628986 us; error 35 us; skew 500 ppm
I20260620 14:47:36.629396 3336 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 14:47:36 dist-test-slave-q9j6 krb5kdc[3323](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781966856, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 14:47:36.639011 3336 init.cc:377] Logged in from keytab as kudu/127.0.97.62@KRBTEST.COM (short username kudu)
I20260620 14:47:36.640709 3336 webserver.cc:460] Webserver started at http://127.0.97.62:36791/ using document root <none> and password file <none>
I20260620 14:47:36.641461 3336 fs_manager.cc:359] Metadata directory not provided
I20260620 14:47:36.641595 3336 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 14:47:36.641927 3336 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 14:47:36.645150 3336 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskUci3DU/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1781966829091300-388-0/minicluster-data/master-0/data/instance:
uuid: "4dc98c2742b34e73a12d300816398e5d"
format_stamp: "Formatted at 2026-06-20 14:47:36 on dist-test-slave-q9j6"
server_key: "22b38615234e3728c4937b3e57f9d505"
server_key_iv: "6afa546799f506880a445eb1468b7878"
server_key_version: "encryptionkey@0"
I20260620 14:47:36.645988 3336 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskUci3DU/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1781966829091300-388-0/minicluster-data/master-0/wal/instance:
uuid: "4dc98c2742b34e73a12d300816398e5d"
format_stamp: "Formatted at 2026-06-20 14:47:36 on dist-test-slave-q9j6"
server_key: "22b38615234e3728c4937b3e57f9d505"
server_key_iv: "6afa546799f506880a445eb1468b7878"
server_key_version: "encryptionkey@0"
I20260620 14:47:36.651392 3336 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.007s sys 0.000s
I20260620 14:47:36.654924 3351 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 14:47:36.656287 3336 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260620 14:47:36.656497 3336 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskUci3DU/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1781966829091300-388-0/minicluster-data/master-0/data,/tmp/dist-test-taskUci3DU/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1781966829091300-388-0/minicluster-data/master-0/wal
uuid: "4dc98c2742b34e73a12d300816398e5d"
format_stamp: "Formatted at 2026-06-20 14:47:36 on dist-test-slave-q9j6"
server_key: "22b38615234e3728c4937b3e57f9d505"
server_key_iv: "6afa546799f506880a445eb1468b7878"
server_key_version: "encryptionkey@0"
I20260620 14:47:36.656709 3336 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskUci3DU/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1781966829091300-388-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskUci3DU/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1781966829091300-388-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskUci3DU/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1781966829091300-388-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 14:47:36.693362 3336 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 14:47:36.697746 3336 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 14:47:36.698060 3336 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 14:47:36.716897 3336 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.62:44827
I20260620 14:47:36.716917 3402 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.62:44827 every 8 connection(s)
I20260620 14:47:36.718371 3336 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskUci3DU/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1781966829091300-388-0/minicluster-data/master-0/data/info.pb
I20260620 14:47:36.722227 3403 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 14:47:36.722685 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskUci3DU/build/asan/bin/kudu as pid 3336
I20260620 14:47:36.722903 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskUci3DU/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1781966829091300-388-0/minicluster-data/master-0/wal/instance
I20260620 14:47:36.723542 388 external_mini_cluster.cc:1428] Setting key 0899ac3f09641d02eeb951147dd3ff2f
I20260620 14:47:36.736378 3403 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 4dc98c2742b34e73a12d300816398e5d: Bootstrap starting.
Jun 20 14:47:36 dist-test-slave-q9j6 krb5kdc[3323](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781966856, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260620 14:47:36.740217 3403 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 4dc98c2742b34e73a12d300816398e5d: Neither blocks nor log segments found. Creating new log.
I20260620 14:47:36.741304 3403 log.cc:824] T 00000000000000000000000000000000 P 4dc98c2742b34e73a12d300816398e5d: Log is configured to *not* fsync() on all Append() calls
I20260620 14:47:36.744678 3403 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 4dc98c2742b34e73a12d300816398e5d: No bootstrap required, opened a new log
I20260620 14:47:36.747815 3406 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:47:36.725554 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55632 (local address 127.0.97.62:44827)
0620 14:47:36.726380 (+ 826us) server_negotiation.cc:206] Beginning negotiation
0620 14:47:36.726396 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0620 14:47:36.726444 (+ 48us) server_negotiation.cc:407] Connection header received
0620 14:47:36.728220 (+ 1776us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:47:36.728246 (+ 26us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:47:36.728702 (+ 456us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:47:36.729053 (+ 351us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:47:36.730880 (+ 1827us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:47:36.732275 (+ 1395us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:47:36.734200 (+ 1925us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:47:36.734936 (+ 736us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:47:36.735075 (+ 139us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:47:36.739745 (+ 4670us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 14:47:36.739754 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 14:47:36.739771 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 14:47:36.739818 (+ 47us) server_negotiation.cc:922] calling sasl_server_start()
0620 14:47:36.743275 (+ 3457us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:47:36.744041 (+ 766us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:47:36.744049 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:47:36.744060 (+ 11us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:47:36.744254 (+ 194us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:47:36.744601 (+ 347us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:47:36.744606 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:47:36.744609 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:47:36.745338 (+ 729us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 14:47:36.745649 (+ 311us) server_negotiation.cc:1035] Waiting for connection context
0620 14:47:36.745945 (+ 296us) server_negotiation.cc:299] Negotiation successful
0620 14:47:36.746394 (+ 449us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":584,"thread_start_us":373,"threads_started":1}
I20260620 14:47:36.751788 388 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260620 14:47:36.753813 3403 raft_consensus.cc:348] T 00000000000000000000000000000000 P 4dc98c2742b34e73a12d300816398e5d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4dc98c2742b34e73a12d300816398e5d" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 44827 } }
I20260620 14:47:36.754184 3403 raft_consensus.cc:374] T 00000000000000000000000000000000 P 4dc98c2742b34e73a12d300816398e5d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 14:47:36.754302 3403 raft_consensus.cc:729] T 00000000000000000000000000000000 P 4dc98c2742b34e73a12d300816398e5d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4dc98c2742b34e73a12d300816398e5d, State: Initialized, Role: FOLLOWER
I20260620 14:47:36.754870 3403 consensus_queue.cc:260] T 00000000000000000000000000000000 P 4dc98c2742b34e73a12d300816398e5d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4dc98c2742b34e73a12d300816398e5d" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 44827 } }
I20260620 14:47:36.755087 3403 raft_consensus.cc:388] T 00000000000000000000000000000000 P 4dc98c2742b34e73a12d300816398e5d [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 14:47:36.755215 3403 raft_consensus.cc:482] T 00000000000000000000000000000000 P 4dc98c2742b34e73a12d300816398e5d [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 14:47:36.755390 3403 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 4dc98c2742b34e73a12d300816398e5d [term 0 FOLLOWER]: Advancing to term 1
I20260620 14:47:36.757541 3403 raft_consensus.cc:504] T 00000000000000000000000000000000 P 4dc98c2742b34e73a12d300816398e5d [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4dc98c2742b34e73a12d300816398e5d" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 44827 } }
I20260620 14:47:36.758046 3403 leader_election.cc:302] T 00000000000000000000000000000000 P 4dc98c2742b34e73a12d300816398e5d [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 4dc98c2742b34e73a12d300816398e5d; no voters:
I20260620 14:47:36.758680 3403 leader_election.cc:288] T 00000000000000000000000000000000 P 4dc98c2742b34e73a12d300816398e5d [CANDIDATE]: Term 1 election: Requested vote from peers
I20260620 14:47:36.758791 3409 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 4dc98c2742b34e73a12d300816398e5d [term 1 FOLLOWER]: Leader election won for term 1
I20260620 14:47:36.761371 3403 sys_catalog.cc:564] T 00000000000000000000000000000000 P 4dc98c2742b34e73a12d300816398e5d [sys.catalog]: configured and running, proceeding with master startup.
I20260620 14:47:36.764503 3409 raft_consensus.cc:686] T 00000000000000000000000000000000 P 4dc98c2742b34e73a12d300816398e5d [term 1 LEADER]: Becoming Leader. State: Replica: 4dc98c2742b34e73a12d300816398e5d, State: Running, Role: LEADER
I20260620 14:47:36.765158 3409 consensus_queue.cc:237] T 00000000000000000000000000000000 P 4dc98c2742b34e73a12d300816398e5d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4dc98c2742b34e73a12d300816398e5d" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 44827 } }
I20260620 14:47:36.768427 3410 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4dc98c2742b34e73a12d300816398e5d [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "4dc98c2742b34e73a12d300816398e5d" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4dc98c2742b34e73a12d300816398e5d" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 44827 } } }
I20260620 14:47:36.768858 3410 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4dc98c2742b34e73a12d300816398e5d [sys.catalog]: This master's current role is: LEADER
I20260620 14:47:36.768695 3411 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4dc98c2742b34e73a12d300816398e5d [sys.catalog]: SysCatalogTable state changed. Reason: New leader 4dc98c2742b34e73a12d300816398e5d. Latest consensus state: current_term: 1 leader_uuid: "4dc98c2742b34e73a12d300816398e5d" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4dc98c2742b34e73a12d300816398e5d" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 44827 } } }
I20260620 14:47:36.768970 3411 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4dc98c2742b34e73a12d300816398e5d [sys.catalog]: This master's current role is: LEADER
I20260620 14:47:36.774667 3423 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 14:47:36.785115 3423 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 14:47:36.794598 3423 catalog_manager.cc:1347] Generated new cluster ID: cc3733d3461d49369453e891a3f843dd
I20260620 14:47:36.794787 3423 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 14:47:36.797415 3406 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:47:36.788563 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55636 (local address 127.0.97.62:44827)
0620 14:47:36.788720 (+ 157us) server_negotiation.cc:206] Beginning negotiation
0620 14:47:36.788724 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 14:47:36.789467 (+ 743us) server_negotiation.cc:407] Connection header received
0620 14:47:36.789672 (+ 205us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:47:36.789679 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:47:36.789773 (+ 94us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:47:36.789884 (+ 111us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:47:36.791153 (+ 1269us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:47:36.792278 (+ 1125us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:47:36.793603 (+ 1325us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:47:36.794094 (+ 491us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:47:36.794213 (+ 119us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:47:36.795724 (+ 1511us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 14:47:36.795733 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 14:47:36.795741 (+ 8us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 14:47:36.796564 (+ 823us) server_negotiation.cc:922] calling sasl_server_start()
0620 14:47:36.796648 (+ 84us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 14:47:36.796688 (+ 40us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 14:47:36.796802 (+ 114us) server_negotiation.cc:1035] Waiting for connection context
0620 14:47:36.796965 (+ 163us) server_negotiation.cc:299] Negotiation successful
0620 14:47:36.797069 (+ 104us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":40}
I20260620 14:47:36.811250 3423 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 14:47:36.812837 3423 catalog_manager.cc:1504] Loading token signing keys...
I20260620 14:47:36.819674 3423 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 4dc98c2742b34e73a12d300816398e5d: Generated new TSK 0
I20260620 14:47:36.820574 3423 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260620 14:47:36.836302 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskUci3DU/build/asan/bin/kudu pid 3336
I20260620 14:47:36.837551 3398 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 14:47:36.929394 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskUci3DU/build/asan/bin/kudu with pid 3336
2026-06-20T14:47:36Z chronyd exiting
[ OK ] SecurityITest.SaslPlainFallback (1975 ms)
[ RUN ] SecurityITest.TestUnauthorizedClientKerberosCredentials
Loading random data
Initializing database '/tmp/dist-test-taskUci3DU/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781966829091300-388-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 20 14:47:37 dist-test-slave-q9j6 krb5kdc[3437](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 20 14:47:37 dist-test-slave-q9j6 krb5kdc[3437](info): set up 2 sockets
Jun 20 14:47:37 dist-test-slave-q9j6 krb5kdc[3437](info): commencing operation
krb5kdc: starting...
W20260620 14:47:38.012140 388 mini_kdc.cc:121] Time spent starting KDC: real 1.045s user 0.001s sys 0.016s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskUci3DU/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781966829091300-388-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskUci3DU/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781966829091300-388-0/krb5kdc/test-user.keytab.
Jun 20 14:47:38 dist-test-slave-q9j6 krb5kdc[3437](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781966858, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-20T14:47:38Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T14:47:38Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.97.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskUci3DU/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781966829091300-388-0/krb5kdc/kudu_127.0.97.62.keytab.
Entry for principal kudu/127.0.97.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskUci3DU/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781966829091300-388-0/krb5kdc/kudu_127.0.97.62.keytab.
I20260620 14:47:38.205367 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskUci3DU/build/asan/bin/kudu
/tmp/dist-test-taskUci3DU/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskUci3DU/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781966829091300-388-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskUci3DU/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781966829091300-388-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskUci3DU/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781966829091300-388-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskUci3DU/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781966829091300-388-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:38725
--webserver_interface=127.0.97.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskUci3DU/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781966829091300-388-0/krb5kdc/kudu_127.0.97.62.keytab
--principal=kudu/127.0.97.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:43973
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.97.62:38725
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskUci3DU/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781966829091300-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskUci3DU/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781966829091300-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskUci3DU/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781966829091300-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 14:47:38.526769 3450 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 14:47:38.527144 3450 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 14:47:38.527311 3450 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 14:47:38.534577 3450 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 14:47:38.534746 3450 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 14:47:38.534861 3450 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 14:47:38.534966 3450 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 14:47:38.535043 3450 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 14:47:38.545202 3450 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:43973
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskUci3DU/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781966829091300-388-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskUci3DU/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781966829091300-388-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.97.62:38725
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskUci3DU/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781966829091300-388-0/krb5kdc/kudu_127.0.97.62.keytab
--principal=kudu/127.0.97.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:38725
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskUci3DU/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781966829091300-388-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.97.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskUci3DU/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781966829091300-388-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 13:41:20 UTC on ccycloud.quasar-vdagoi.root.comops.site
build id 4457462
ASAN enabled
I20260620 14:47:38.546625 3450 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 14:47:38.548046 3450 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 14:47:38.560184 3455 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 14:47:38.560267 3456 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 14:47:38.560319 3458 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 14:47:38.561275 3450 server_base.cc:1034] running on GCE node
I20260620 14:47:38.562177 3450 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 14:47:38.563750 3450 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 14:47:38.564954 3450 hybrid_clock.cc:648] HybridClock initialized: now 1781966858564891 us; error 53 us; skew 500 ppm
I20260620 14:47:38.565274 3450 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 14:47:38 dist-test-slave-q9j6 krb5kdc[3437](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781966858, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 14:47:38.568470 3450 init.cc:377] Logged in from keytab as kudu/127.0.97.62@KRBTEST.COM (short username kudu)
I20260620 14:47:38.569989 3450 webserver.cc:460] Webserver started at http://127.0.97.62:32931/ using document root <none> and password file <none>
I20260620 14:47:38.570726 3450 fs_manager.cc:359] Metadata directory not provided
I20260620 14:47:38.570838 3450 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 14:47:38.571103 3450 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 14:47:38.574138 3450 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskUci3DU/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781966829091300-388-0/minicluster-data/master-0/data/instance:
uuid: "50cb9db8ba4d4a9c9445467fc8d572d1"
format_stamp: "Formatted at 2026-06-20 14:47:38 on dist-test-slave-q9j6"
server_key: "ba6da5f1b881b07dc3ef8d6e19a94ede"
server_key_iv: "24fefcd64f51f5b5f84114b5a464104d"
server_key_version: "encryptionkey@0"
I20260620 14:47:38.574862 3450 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskUci3DU/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781966829091300-388-0/minicluster-data/master-0/wal/instance:
uuid: "50cb9db8ba4d4a9c9445467fc8d572d1"
format_stamp: "Formatted at 2026-06-20 14:47:38 on dist-test-slave-q9j6"
server_key: "ba6da5f1b881b07dc3ef8d6e19a94ede"
server_key_iv: "24fefcd64f51f5b5f84114b5a464104d"
server_key_version: "encryptionkey@0"
I20260620 14:47:38.580144 3450 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.002s
I20260620 14:47:38.583757 3465 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 14:47:38.585214 3450 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260620 14:47:38.585433 3450 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskUci3DU/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781966829091300-388-0/minicluster-data/master-0/data,/tmp/dist-test-taskUci3DU/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781966829091300-388-0/minicluster-data/master-0/wal
uuid: "50cb9db8ba4d4a9c9445467fc8d572d1"
format_stamp: "Formatted at 2026-06-20 14:47:38 on dist-test-slave-q9j6"
server_key: "ba6da5f1b881b07dc3ef8d6e19a94ede"
server_key_iv: "24fefcd64f51f5b5f84114b5a464104d"
server_key_version: "encryptionkey@0"
I20260620 14:47:38.585652 3450 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskUci3DU/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781966829091300-388-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskUci3DU/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781966829091300-388-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskUci3DU/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781966829091300-388-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 14:47:38.602005 3450 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 14:47:38.606503 3450 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 14:47:38.606786 3450 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 14:47:38.629483 3450 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.62:38725
I20260620 14:47:38.629511 3516 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.62:38725 every 8 connection(s)
I20260620 14:47:38.630970 3450 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskUci3DU/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781966829091300-388-0/minicluster-data/master-0/data/info.pb
I20260620 14:47:38.633960 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskUci3DU/build/asan/bin/kudu as pid 3450
I20260620 14:47:38.634207 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskUci3DU/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781966829091300-388-0/minicluster-data/master-0/wal/instance
I20260620 14:47:38.634995 388 external_mini_cluster.cc:1428] Setting key 90478fdb92ab9a57e9c5a744338364f4
I20260620 14:47:38.635398 3517 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 14:47:38.650252 3517 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 50cb9db8ba4d4a9c9445467fc8d572d1: Bootstrap starting.
Jun 20 14:47:38 dist-test-slave-q9j6 krb5kdc[3437](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781966858, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260620 14:47:38.653472 3517 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 50cb9db8ba4d4a9c9445467fc8d572d1: Neither blocks nor log segments found. Creating new log.
I20260620 14:47:38.654518 3517 log.cc:824] T 00000000000000000000000000000000 P 50cb9db8ba4d4a9c9445467fc8d572d1: Log is configured to *not* fsync() on all Append() calls
I20260620 14:47:38.657486 3517 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 50cb9db8ba4d4a9c9445467fc8d572d1: No bootstrap required, opened a new log
I20260620 14:47:38.658987 3520 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:47:38.637088 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53404 (local address 127.0.97.62:38725)
0620 14:47:38.637851 (+ 763us) server_negotiation.cc:206] Beginning negotiation
0620 14:47:38.637862 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0620 14:47:38.637903 (+ 41us) server_negotiation.cc:407] Connection header received
0620 14:47:38.640328 (+ 2425us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:47:38.640353 (+ 25us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:47:38.640781 (+ 428us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:47:38.641140 (+ 359us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:47:38.642879 (+ 1739us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:47:38.644277 (+ 1398us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:47:38.646032 (+ 1755us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:47:38.646771 (+ 739us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:47:38.646945 (+ 174us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:47:38.651654 (+ 4709us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 14:47:38.651665 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 14:47:38.651692 (+ 27us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 14:47:38.651736 (+ 44us) server_negotiation.cc:922] calling sasl_server_start()
0620 14:47:38.654835 (+ 3099us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:47:38.655573 (+ 738us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:47:38.655579 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:47:38.655586 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:47:38.655817 (+ 231us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:47:38.656211 (+ 394us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:47:38.656223 (+ 12us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:47:38.656225 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:47:38.656752 (+ 527us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 14:47:38.657077 (+ 325us) server_negotiation.cc:1035] Waiting for connection context
0620 14:47:38.657519 (+ 442us) server_negotiation.cc:299] Negotiation successful
0620 14:47:38.657950 (+ 431us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":564,"thread_start_us":352,"threads_started":1}
I20260620 14:47:38.666554 3517 raft_consensus.cc:348] T 00000000000000000000000000000000 P 50cb9db8ba4d4a9c9445467fc8d572d1 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "50cb9db8ba4d4a9c9445467fc8d572d1" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 38725 } }
I20260620 14:47:38.666980 3517 raft_consensus.cc:374] T 00000000000000000000000000000000 P 50cb9db8ba4d4a9c9445467fc8d572d1 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 14:47:38.667093 3517 raft_consensus.cc:729] T 00000000000000000000000000000000 P 50cb9db8ba4d4a9c9445467fc8d572d1 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 50cb9db8ba4d4a9c9445467fc8d572d1, State: Initialized, Role: FOLLOWER
I20260620 14:47:38.667809 3517 consensus_queue.cc:260] T 00000000000000000000000000000000 P 50cb9db8ba4d4a9c9445467fc8d572d1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "50cb9db8ba4d4a9c9445467fc8d572d1" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 38725 } }
I20260620 14:47:38.668044 3517 raft_consensus.cc:388] T 00000000000000000000000000000000 P 50cb9db8ba4d4a9c9445467fc8d572d1 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 14:47:38.668190 3517 raft_consensus.cc:482] T 00000000000000000000000000000000 P 50cb9db8ba4d4a9c9445467fc8d572d1 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 14:47:38.668370 3517 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 50cb9db8ba4d4a9c9445467fc8d572d1 [term 0 FOLLOWER]: Advancing to term 1
I20260620 14:47:38.670720 3517 raft_consensus.cc:504] T 00000000000000000000000000000000 P 50cb9db8ba4d4a9c9445467fc8d572d1 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "50cb9db8ba4d4a9c9445467fc8d572d1" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 38725 } }
I20260620 14:47:38.671231 3517 leader_election.cc:302] T 00000000000000000000000000000000 P 50cb9db8ba4d4a9c9445467fc8d572d1 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 50cb9db8ba4d4a9c9445467fc8d572d1; no voters:
I20260620 14:47:38.671849 3517 leader_election.cc:288] T 00000000000000000000000000000000 P 50cb9db8ba4d4a9c9445467fc8d572d1 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260620 14:47:38.672003 3522 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 50cb9db8ba4d4a9c9445467fc8d572d1 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 14:47:38.672715 3522 raft_consensus.cc:686] T 00000000000000000000000000000000 P 50cb9db8ba4d4a9c9445467fc8d572d1 [term 1 LEADER]: Becoming Leader. State: Replica: 50cb9db8ba4d4a9c9445467fc8d572d1, State: Running, Role: LEADER
I20260620 14:47:38.673286 3522 consensus_queue.cc:237] T 00000000000000000000000000000000 P 50cb9db8ba4d4a9c9445467fc8d572d1 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "50cb9db8ba4d4a9c9445467fc8d572d1" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 38725 } }
I20260620 14:47:38.674108 3517 sys_catalog.cc:564] T 00000000000000000000000000000000 P 50cb9db8ba4d4a9c9445467fc8d572d1 [sys.catalog]: configured and running, proceeding with master startup.
I20260620 14:47:38.676573 3524 sys_catalog.cc:455] T 00000000000000000000000000000000 P 50cb9db8ba4d4a9c9445467fc8d572d1 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 50cb9db8ba4d4a9c9445467fc8d572d1. Latest consensus state: current_term: 1 leader_uuid: "50cb9db8ba4d4a9c9445467fc8d572d1" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "50cb9db8ba4d4a9c9445467fc8d572d1" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 38725 } } }
I20260620 14:47:38.676539 3523 sys_catalog.cc:455] T 00000000000000000000000000000000 P 50cb9db8ba4d4a9c9445467fc8d572d1 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "50cb9db8ba4d4a9c9445467fc8d572d1" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "50cb9db8ba4d4a9c9445467fc8d572d1" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 38725 } } }
I20260620 14:47:38.676949 3524 sys_catalog.cc:458] T 00000000000000000000000000000000 P 50cb9db8ba4d4a9c9445467fc8d572d1 [sys.catalog]: This master's current role is: LEADER
I20260620 14:47:38.676949 3523 sys_catalog.cc:458] T 00000000000000000000000000000000 P 50cb9db8ba4d4a9c9445467fc8d572d1 [sys.catalog]: This master's current role is: LEADER
I20260620 14:47:38.679171 3530 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 14:47:38.684417 3530 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 14:47:38.694175 3530 catalog_manager.cc:1347] Generated new cluster ID: 3376abe195914105a7c51d734f907f12
I20260620 14:47:38.694295 3530 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 14:47:38.714447 3530 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 14:47:38.715924 3530 catalog_manager.cc:1504] Loading token signing keys...
I20260620 14:47:38.724985 3530 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 50cb9db8ba4d4a9c9445467fc8d572d1: Generated new TSK 0
I20260620 14:47:38.725927 3530 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.0.97.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskUci3DU/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781966829091300-388-0/krb5kdc/kudu_127.0.97.1.keytab.
Entry for principal kudu/127.0.97.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskUci3DU/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781966829091300-388-0/krb5kdc/kudu_127.0.97.1.keytab.
I20260620 14:47:38.775039 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskUci3DU/build/asan/bin/kudu
/tmp/dist-test-taskUci3DU/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskUci3DU/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781966829091300-388-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskUci3DU/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781966829091300-388-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskUci3DU/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781966829091300-388-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskUci3DU/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781966829091300-388-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.1:0
--local_ip_for_outbound_sockets=127.0.97.1
--webserver_interface=127.0.97.1
--webserver_port=0
--tserver_master_addrs=127.0.97.62:38725
--keytab_file=/tmp/dist-test-taskUci3DU/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781966829091300-388-0/krb5kdc/kudu_127.0.97.1.keytab
--principal=kudu/127.0.97.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:43973
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskUci3DU/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781966829091300-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskUci3DU/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781966829091300-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskUci3DU/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781966829091300-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 14:47:39.108141 3543 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 14:47:39.108520 3543 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 14:47:39.108690 3543 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 14:47:39.116180 3543 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 14:47:39.116366 3543 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 14:47:39.116556 3543 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.1
I20260620 14:47:39.126654 3543 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:43973
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskUci3DU/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781966829091300-388-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskUci3DU/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781966829091300-388-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskUci3DU/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781966829091300-388-0/krb5kdc/kudu_127.0.97.1.keytab
--principal=kudu/127.0.97.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskUci3DU/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781966829091300-388-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.97.1
--webserver_port=0
--tserver_master_addrs=127.0.97.62:38725
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.1
--log_dir=/tmp/dist-test-taskUci3DU/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781966829091300-388-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 13:41:20 UTC on ccycloud.quasar-vdagoi.root.comops.site
build id 4457462
ASAN enabled
I20260620 14:47:39.128113 3543 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 14:47:39.129774 3543 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 14:47:39.143723 3548 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 14:47:39.143743 3549 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 14:47:39.144009 3551 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 14:47:39.144794 3543 server_base.cc:1034] running on GCE node
I20260620 14:47:39.145401 3543 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 14:47:39.146354 3543 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 14:47:39.147567 3543 hybrid_clock.cc:648] HybridClock initialized: now 1781966859147495 us; error 61 us; skew 500 ppm
I20260620 14:47:39.147974 3543 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 14:47:39 dist-test-slave-q9j6 krb5kdc[3437](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781966859, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 14:47:39.151235 3543 init.cc:377] Logged in from keytab as kudu/127.0.97.1@KRBTEST.COM (short username kudu)
I20260620 14:47:39.152908 3543 webserver.cc:460] Webserver started at http://127.0.97.1:44457/ using document root <none> and password file <none>
I20260620 14:47:39.153667 3543 fs_manager.cc:359] Metadata directory not provided
I20260620 14:47:39.153776 3543 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 14:47:39.154134 3543 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 14:47:39.157337 3543 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskUci3DU/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781966829091300-388-0/minicluster-data/ts-0/data/instance:
uuid: "eaf7e385445942bd960bb9d3e27084fc"
format_stamp: "Formatted at 2026-06-20 14:47:39 on dist-test-slave-q9j6"
server_key: "800b71b5d27423e1376ace3e7d9590a9"
server_key_iv: "303be18bd5cb6d276b10b0e77d60b4c3"
server_key_version: "encryptionkey@0"
I20260620 14:47:39.158167 3543 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskUci3DU/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781966829091300-388-0/minicluster-data/ts-0/wal/instance:
uuid: "eaf7e385445942bd960bb9d3e27084fc"
format_stamp: "Formatted at 2026-06-20 14:47:39 on dist-test-slave-q9j6"
server_key: "800b71b5d27423e1376ace3e7d9590a9"
server_key_iv: "303be18bd5cb6d276b10b0e77d60b4c3"
server_key_version: "encryptionkey@0"
I20260620 14:47:39.163484 3543 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.007s sys 0.000s
I20260620 14:47:39.167073 3558 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 14:47:39.168509 3543 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.003s
I20260620 14:47:39.168696 3543 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskUci3DU/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781966829091300-388-0/minicluster-data/ts-0/data,/tmp/dist-test-taskUci3DU/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781966829091300-388-0/minicluster-data/ts-0/wal
uuid: "eaf7e385445942bd960bb9d3e27084fc"
format_stamp: "Formatted at 2026-06-20 14:47:39 on dist-test-slave-q9j6"
server_key: "800b71b5d27423e1376ace3e7d9590a9"
server_key_iv: "303be18bd5cb6d276b10b0e77d60b4c3"
server_key_version: "encryptionkey@0"
I20260620 14:47:39.168900 3543 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskUci3DU/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781966829091300-388-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskUci3DU/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781966829091300-388-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskUci3DU/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781966829091300-388-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 14:47:39.180647 3543 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 14:47:39.184556 3543 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 14:47:39.184857 3543 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 14:47:39.185817 3543 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 14:47:39.187275 3543 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 14:47:39.187383 3543 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 14:47:39.187546 3543 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 14:47:39.187642 3543 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 14:47:39.221022 3543 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.1:35517
I20260620 14:47:39.221065 3670 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.1:35517 every 8 connection(s)
I20260620 14:47:39.222371 3543 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskUci3DU/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781966829091300-388-0/minicluster-data/ts-0/data/info.pb
I20260620 14:47:39.225558 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskUci3DU/build/asan/bin/kudu as pid 3543
I20260620 14:47:39.225782 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskUci3DU/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781966829091300-388-0/minicluster-data/ts-0/wal/instance
I20260620 14:47:39.226621 388 external_mini_cluster.cc:1428] Setting key aa215b9ff85e09cb1d40e41457bfba83
Jun 20 14:47:39 dist-test-slave-q9j6 krb5kdc[3437](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781966859, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.1@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260620 14:47:39.244943 3673 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:47:39.225004 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.1:41731 (local address 127.0.97.62:38725)
0620 14:47:39.225567 (+ 563us) server_negotiation.cc:206] Beginning negotiation
0620 14:47:39.225574 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0620 14:47:39.226806 (+ 1232us) server_negotiation.cc:407] Connection header received
0620 14:47:39.228827 (+ 2021us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:47:39.228831 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:47:39.228898 (+ 67us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:47:39.228991 (+ 93us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:47:39.231226 (+ 2235us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:47:39.231985 (+ 759us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:47:39.234543 (+ 2558us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:47:39.235253 (+ 710us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:47:39.235393 (+ 140us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:47:39.239048 (+ 3655us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 14:47:39.239053 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 14:47:39.239056 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 14:47:39.239088 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0620 14:47:39.241624 (+ 2536us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:47:39.242629 (+ 1005us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:47:39.242633 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:47:39.242635 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:47:39.242743 (+ 108us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:47:39.243248 (+ 505us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:47:39.243251 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:47:39.243253 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:47:39.243515 (+ 262us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 14:47:39.243687 (+ 172us) server_negotiation.cc:1035] Waiting for connection context
0620 14:47:39.244535 (+ 848us) server_negotiation.cc:299] Negotiation successful
0620 14:47:39.244695 (+ 160us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":440,"thread_start_us":335,"threads_started":1}
WARNING: no policy specified for kudu/127.0.97.2@KRBTEST.COM; defaulting to no policy
I20260620 14:47:39.246564 3674 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:47:39.225694 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:38725 (local address 127.0.97.1:41731)
0620 14:47:39.226624 (+ 930us) negotiation.cc:107] Waiting for socket to connect
0620 14:47:39.226681 (+ 57us) client_negotiation.cc:174] Beginning negotiation
0620 14:47:39.228500 (+ 1819us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 14:47:39.229267 (+ 767us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 14:47:39.229284 (+ 17us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 14:47:39.230110 (+ 826us) client_negotiation.cc:189] Negotiated authn=SASL
0620 14:47:39.231033 (+ 923us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:47:39.231063 (+ 30us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:47:39.232142 (+ 1079us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:47:39.232150 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:47:39.234372 (+ 2222us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:47:39.234395 (+ 23us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:47:39.235481 (+ 1086us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:47:39.235486 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:47:39.235757 (+ 271us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:47:39.236466 (+ 709us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 14:47:39.236489 (+ 23us) client_negotiation.cc:656] calling sasl_client_start()
0620 14:47:39.238809 (+ 2320us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 14:47:39.241809 (+ 3000us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 14:47:39.241822 (+ 13us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 14:47:39.241845 (+ 23us) client_negotiation.cc:760] calling sasl_client_step()
0620 14:47:39.242512 (+ 667us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 14:47:39.242855 (+ 343us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 14:47:39.242862 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 14:47:39.242870 (+ 8us) client_negotiation.cc:760] calling sasl_client_step()
0620 14:47:39.243136 (+ 266us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 14:47:39.243676 (+ 540us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 14:47:39.243688 (+ 12us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 14:47:39.244142 (+ 454us) client_negotiation.cc:769] Sending connection context
0620 14:47:39.244483 (+ 341us) client_negotiation.cc:240] Negotiation successful
0620 14:47:39.245061 (+ 578us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":583,"thread_start_us":327,"threads_started":1}
I20260620 14:47:39.248251 3671 heartbeater.cc:347] Connected to a master server at 127.0.97.62:38725
I20260620 14:47:39.248520 3671 heartbeater.cc:464] Registering TS with master...
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.2@KRBTEST.COM" created.
I20260620 14:47:39.249363 3671 heartbeater.cc:511] Master 127.0.97.62:38725 requested a full tablet report, sending...
I20260620 14:47:39.251518 3482 ts_manager.cc:194] Registered new tserver with Master: eaf7e385445942bd960bb9d3e27084fc (127.0.97.1:35517)
I20260620 14:47:39.253003 3482 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.97.1@KRBTEST.COM'} at 127.0.97.1:41731
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskUci3DU/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781966829091300-388-0/krb5kdc/kudu_127.0.97.2.keytab.
Entry for principal kudu/127.0.97.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskUci3DU/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781966829091300-388-0/krb5kdc/kudu_127.0.97.2.keytab.
I20260620 14:47:39.270819 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskUci3DU/build/asan/bin/kudu
/tmp/dist-test-taskUci3DU/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskUci3DU/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781966829091300-388-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskUci3DU/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781966829091300-388-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskUci3DU/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781966829091300-388-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskUci3DU/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781966829091300-388-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.2:0
--local_ip_for_outbound_sockets=127.0.97.2
--webserver_interface=127.0.97.2
--webserver_port=0
--tserver_master_addrs=127.0.97.62:38725
--keytab_file=/tmp/dist-test-taskUci3DU/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781966829091300-388-0/krb5kdc/kudu_127.0.97.2.keytab
--principal=kudu/127.0.97.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:43973
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskUci3DU/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781966829091300-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskUci3DU/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781966829091300-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskUci3DU/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781966829091300-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 14:47:39.597010 3677 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 14:47:39.597288 3677 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 14:47:39.597440 3677 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 14:47:39.604701 3677 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 14:47:39.604820 3677 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 14:47:39.605011 3677 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.2
I20260620 14:47:39.615134 3677 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:43973
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskUci3DU/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781966829091300-388-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskUci3DU/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781966829091300-388-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskUci3DU/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781966829091300-388-0/krb5kdc/kudu_127.0.97.2.keytab
--principal=kudu/127.0.97.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskUci3DU/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781966829091300-388-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.97.2
--webserver_port=0
--tserver_master_addrs=127.0.97.62:38725
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.2
--log_dir=/tmp/dist-test-taskUci3DU/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781966829091300-388-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 13:41:20 UTC on ccycloud.quasar-vdagoi.root.comops.site
build id 4457462
ASAN enabled
I20260620 14:47:39.616592 3677 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 14:47:39.617942 3677 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 14:47:39.631556 3683 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 14:47:39.631831 3682 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 14:47:39.631533 3685 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 14:47:39.632297 3677 server_base.cc:1034] running on GCE node
I20260620 14:47:39.632923 3677 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 14:47:39.633905 3677 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 14:47:39.635139 3677 hybrid_clock.cc:648] HybridClock initialized: now 1781966859635086 us; error 49 us; skew 500 ppm
I20260620 14:47:39.635481 3677 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 14:47:39 dist-test-slave-q9j6 krb5kdc[3437](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781966859, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 14:47:39.638731 3677 init.cc:377] Logged in from keytab as kudu/127.0.97.2@KRBTEST.COM (short username kudu)
I20260620 14:47:39.640419 3677 webserver.cc:460] Webserver started at http://127.0.97.2:35091/ using document root <none> and password file <none>
I20260620 14:47:39.641152 3677 fs_manager.cc:359] Metadata directory not provided
I20260620 14:47:39.641279 3677 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 14:47:39.641587 3677 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 14:47:39.644696 3677 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskUci3DU/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781966829091300-388-0/minicluster-data/ts-1/data/instance:
uuid: "58414156a609424db4df4588649b336c"
format_stamp: "Formatted at 2026-06-20 14:47:39 on dist-test-slave-q9j6"
server_key: "ebe2a55e068d0732182371a241bdfb95"
server_key_iv: "f262c6a06f19ecda58384d6ba2f7992f"
server_key_version: "encryptionkey@0"
I20260620 14:47:39.645524 3677 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskUci3DU/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781966829091300-388-0/minicluster-data/ts-1/wal/instance:
uuid: "58414156a609424db4df4588649b336c"
format_stamp: "Formatted at 2026-06-20 14:47:39 on dist-test-slave-q9j6"
server_key: "ebe2a55e068d0732182371a241bdfb95"
server_key_iv: "f262c6a06f19ecda58384d6ba2f7992f"
server_key_version: "encryptionkey@0"
I20260620 14:47:39.650897 3677 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.000s
I20260620 14:47:39.654626 3692 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 14:47:39.656134 3677 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260620 14:47:39.656320 3677 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskUci3DU/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781966829091300-388-0/minicluster-data/ts-1/data,/tmp/dist-test-taskUci3DU/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781966829091300-388-0/minicluster-data/ts-1/wal
uuid: "58414156a609424db4df4588649b336c"
format_stamp: "Formatted at 2026-06-20 14:47:39 on dist-test-slave-q9j6"
server_key: "ebe2a55e068d0732182371a241bdfb95"
server_key_iv: "f262c6a06f19ecda58384d6ba2f7992f"
server_key_version: "encryptionkey@0"
I20260620 14:47:39.656539 3677 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskUci3DU/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781966829091300-388-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskUci3DU/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781966829091300-388-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskUci3DU/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781966829091300-388-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 14:47:39.673367 3677 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 14:47:39.678334 3677 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 14:47:39.678630 3677 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 14:47:39.679693 3677 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 14:47:39.681054 3677 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 14:47:39.681129 3677 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 14:47:39.681317 3677 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 14:47:39.681367 3677 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 14:47:39.714403 3677 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.2:35741
I20260620 14:47:39.714445 3804 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.2:35741 every 8 connection(s)
I20260620 14:47:39.715770 3677 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskUci3DU/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781966829091300-388-0/minicluster-data/ts-1/data/info.pb
I20260620 14:47:39.720310 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskUci3DU/build/asan/bin/kudu as pid 3677
I20260620 14:47:39.720510 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskUci3DU/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781966829091300-388-0/minicluster-data/ts-1/wal/instance
I20260620 14:47:39.720934 388 external_mini_cluster.cc:1428] Setting key c1c88f742ca72d1832095b886b97d1bf
Jun 20 14:47:39 dist-test-slave-q9j6 krb5kdc[3437](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781966859, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.2@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260620 14:47:39.735976 3673 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:47:39.718292 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.2:52281 (local address 127.0.97.62:38725)
0620 14:47:39.718464 (+ 172us) server_negotiation.cc:206] Beginning negotiation
0620 14:47:39.718468 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 14:47:39.719567 (+ 1099us) server_negotiation.cc:407] Connection header received
0620 14:47:39.721548 (+ 1981us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:47:39.721554 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:47:39.721652 (+ 98us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:47:39.721766 (+ 114us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:47:39.723325 (+ 1559us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:47:39.724185 (+ 860us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:47:39.725976 (+ 1791us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:47:39.726369 (+ 393us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:47:39.726506 (+ 137us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:47:39.730255 (+ 3749us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 14:47:39.730260 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 14:47:39.730264 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 14:47:39.730299 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0620 14:47:39.732837 (+ 2538us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:47:39.733615 (+ 778us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:47:39.733620 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:47:39.733624 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:47:39.733760 (+ 136us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:47:39.734195 (+ 435us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:47:39.734202 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:47:39.734205 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:47:39.734484 (+ 279us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 14:47:39.734633 (+ 149us) server_negotiation.cc:1035] Waiting for connection context
0620 14:47:39.735338 (+ 705us) server_negotiation.cc:299] Negotiation successful
0620 14:47:39.735565 (+ 227us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":56}
I20260620 14:47:39.737017 3807 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:47:39.718700 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:38725 (local address 127.0.97.2:52281)
0620 14:47:39.719394 (+ 694us) negotiation.cc:107] Waiting for socket to connect
0620 14:47:39.719438 (+ 44us) client_negotiation.cc:174] Beginning negotiation
0620 14:47:39.721245 (+ 1807us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 14:47:39.722022 (+ 777us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 14:47:39.722032 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 14:47:39.722578 (+ 546us) client_negotiation.cc:189] Negotiated authn=SASL
0620 14:47:39.723159 (+ 581us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:47:39.723175 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:47:39.724342 (+ 1167us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:47:39.724346 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:47:39.725818 (+ 1472us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:47:39.725829 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:47:39.726597 (+ 768us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:47:39.726604 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:47:39.726843 (+ 239us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:47:39.727556 (+ 713us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 14:47:39.727578 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0620 14:47:39.730009 (+ 2431us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 14:47:39.733010 (+ 3001us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 14:47:39.733019 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 14:47:39.733040 (+ 21us) client_negotiation.cc:760] calling sasl_client_step()
0620 14:47:39.733484 (+ 444us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 14:47:39.733876 (+ 392us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 14:47:39.733882 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 14:47:39.733887 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0620 14:47:39.734083 (+ 196us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 14:47:39.734622 (+ 539us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 14:47:39.734631 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 14:47:39.734989 (+ 358us) client_negotiation.cc:769] Sending connection context
0620 14:47:39.735260 (+ 271us) client_negotiation.cc:240] Negotiation successful
0620 14:47:39.735882 (+ 622us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":456,"thread_start_us":274,"threads_started":1}
WARNING: no policy specified for kudu/127.0.97.3@KRBTEST.COM; defaulting to no policy
I20260620 14:47:39.738605 3805 heartbeater.cc:347] Connected to a master server at 127.0.97.62:38725
I20260620 14:47:39.738934 3805 heartbeater.cc:464] Registering TS with master...
I20260620 14:47:39.739845 3805 heartbeater.cc:511] Master 127.0.97.62:38725 requested a full tablet report, sending...
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.3@KRBTEST.COM" created.
I20260620 14:47:39.741359 3482 ts_manager.cc:194] Registered new tserver with Master: 58414156a609424db4df4588649b336c (127.0.97.2:35741)
I20260620 14:47:39.742158 3482 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.97.2@KRBTEST.COM'} at 127.0.97.2:52281
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskUci3DU/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781966829091300-388-0/krb5kdc/kudu_127.0.97.3.keytab.
Entry for principal kudu/127.0.97.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskUci3DU/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781966829091300-388-0/krb5kdc/kudu_127.0.97.3.keytab.
I20260620 14:47:39.762531 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskUci3DU/build/asan/bin/kudu
/tmp/dist-test-taskUci3DU/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskUci3DU/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781966829091300-388-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskUci3DU/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781966829091300-388-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskUci3DU/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781966829091300-388-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskUci3DU/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781966829091300-388-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.3:0
--local_ip_for_outbound_sockets=127.0.97.3
--webserver_interface=127.0.97.3
--webserver_port=0
--tserver_master_addrs=127.0.97.62:38725
--keytab_file=/tmp/dist-test-taskUci3DU/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781966829091300-388-0/krb5kdc/kudu_127.0.97.3.keytab
--principal=kudu/127.0.97.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:43973
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskUci3DU/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781966829091300-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskUci3DU/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781966829091300-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskUci3DU/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781966829091300-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 14:47:40.095506 3810 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 14:47:40.095911 3810 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 14:47:40.096017 3810 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 14:47:40.103410 3810 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 14:47:40.103538 3810 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 14:47:40.103724 3810 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.3
I20260620 14:47:40.113924 3810 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:43973
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskUci3DU/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781966829091300-388-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskUci3DU/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781966829091300-388-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskUci3DU/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781966829091300-388-0/krb5kdc/kudu_127.0.97.3.keytab
--principal=kudu/127.0.97.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskUci3DU/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781966829091300-388-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.97.3
--webserver_port=0
--tserver_master_addrs=127.0.97.62:38725
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.3
--log_dir=/tmp/dist-test-taskUci3DU/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781966829091300-388-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 13:41:20 UTC on ccycloud.quasar-vdagoi.root.comops.site
build id 4457462
ASAN enabled
I20260620 14:47:40.115234 3810 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 14:47:40.116591 3810 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 14:47:40.129528 3815 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 14:47:40.129552 3816 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 14:47:40.129587 3818 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 14:47:40.130446 3810 server_base.cc:1034] running on GCE node
I20260620 14:47:40.131078 3810 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 14:47:40.132335 3810 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 14:47:40.133567 3810 hybrid_clock.cc:648] HybridClock initialized: now 1781966860133486 us; error 75 us; skew 500 ppm
I20260620 14:47:40.133935 3810 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 14:47:40 dist-test-slave-q9j6 krb5kdc[3437](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781966860, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 14:47:40.137632 3810 init.cc:377] Logged in from keytab as kudu/127.0.97.3@KRBTEST.COM (short username kudu)
I20260620 14:47:40.139171 3810 webserver.cc:460] Webserver started at http://127.0.97.3:45289/ using document root <none> and password file <none>
I20260620 14:47:40.140080 3810 fs_manager.cc:359] Metadata directory not provided
I20260620 14:47:40.140213 3810 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 14:47:40.140553 3810 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 14:47:40.143553 3810 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskUci3DU/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781966829091300-388-0/minicluster-data/ts-2/data/instance:
uuid: "01cc50c3dcbd4f399b3a9fc396c14211"
format_stamp: "Formatted at 2026-06-20 14:47:40 on dist-test-slave-q9j6"
server_key: "1e973b11e280ca5ab9b5c23f6ba15fdc"
server_key_iv: "ea4456bc9c849a2c11de383444cffd90"
server_key_version: "encryptionkey@0"
I20260620 14:47:40.144415 3810 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskUci3DU/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781966829091300-388-0/minicluster-data/ts-2/wal/instance:
uuid: "01cc50c3dcbd4f399b3a9fc396c14211"
format_stamp: "Formatted at 2026-06-20 14:47:40 on dist-test-slave-q9j6"
server_key: "1e973b11e280ca5ab9b5c23f6ba15fdc"
server_key_iv: "ea4456bc9c849a2c11de383444cffd90"
server_key_version: "encryptionkey@0"
I20260620 14:47:40.149576 3810 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.004s sys 0.000s
I20260620 14:47:40.153141 3825 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 14:47:40.154633 3810 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260620 14:47:40.154819 3810 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskUci3DU/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781966829091300-388-0/minicluster-data/ts-2/data,/tmp/dist-test-taskUci3DU/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781966829091300-388-0/minicluster-data/ts-2/wal
uuid: "01cc50c3dcbd4f399b3a9fc396c14211"
format_stamp: "Formatted at 2026-06-20 14:47:40 on dist-test-slave-q9j6"
server_key: "1e973b11e280ca5ab9b5c23f6ba15fdc"
server_key_iv: "ea4456bc9c849a2c11de383444cffd90"
server_key_version: "encryptionkey@0"
I20260620 14:47:40.155032 3810 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskUci3DU/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781966829091300-388-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskUci3DU/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781966829091300-388-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskUci3DU/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781966829091300-388-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 14:47:40.182360 3810 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 14:47:40.186916 3810 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 14:47:40.187275 3810 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 14:47:40.188336 3810 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 14:47:40.189775 3810 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 14:47:40.189875 3810 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 14:47:40.190016 3810 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 14:47:40.190065 3810 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 14:47:40.223405 3810 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.3:38439
I20260620 14:47:40.223446 3937 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.3:38439 every 8 connection(s)
I20260620 14:47:40.224764 3810 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskUci3DU/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781966829091300-388-0/minicluster-data/ts-2/data/info.pb
I20260620 14:47:40.232321 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskUci3DU/build/asan/bin/kudu as pid 3810
I20260620 14:47:40.232548 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskUci3DU/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781966829091300-388-0/minicluster-data/ts-2/wal/instance
I20260620 14:47:40.233016 388 external_mini_cluster.cc:1428] Setting key 34bd113bc8aae070939fe815418b75f6
Jun 20 14:47:40 dist-test-slave-q9j6 krb5kdc[3437](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781966860, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.3@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260620 14:47:40.244833 3673 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:47:40.227530 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.3:56099 (local address 127.0.97.62:38725)
0620 14:47:40.227761 (+ 231us) server_negotiation.cc:206] Beginning negotiation
0620 14:47:40.227766 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 14:47:40.228898 (+ 1132us) server_negotiation.cc:407] Connection header received
0620 14:47:40.230702 (+ 1804us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:47:40.230708 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:47:40.230800 (+ 92us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:47:40.230915 (+ 115us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:47:40.232776 (+ 1861us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:47:40.233685 (+ 909us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:47:40.235810 (+ 2125us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:47:40.236299 (+ 489us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:47:40.236397 (+ 98us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:47:40.239785 (+ 3388us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 14:47:40.239795 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 14:47:40.239798 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 14:47:40.239831 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0620 14:47:40.242200 (+ 2369us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:47:40.242880 (+ 680us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:47:40.242883 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:47:40.242885 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:47:40.243004 (+ 119us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:47:40.243353 (+ 349us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:47:40.243356 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:47:40.243358 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:47:40.243682 (+ 324us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 14:47:40.243786 (+ 104us) server_negotiation.cc:1035] Waiting for connection context
0620 14:47:40.244419 (+ 633us) server_negotiation.cc:299] Negotiation successful
0620 14:47:40.244593 (+ 174us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":106}
I20260620 14:47:40.246225 3940 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:47:40.228022 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:38725 (local address 127.0.97.3:56099)
0620 14:47:40.228723 (+ 701us) negotiation.cc:107] Waiting for socket to connect
0620 14:47:40.228768 (+ 45us) client_negotiation.cc:174] Beginning negotiation
0620 14:47:40.230416 (+ 1648us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 14:47:40.231164 (+ 748us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 14:47:40.231173 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 14:47:40.231753 (+ 580us) client_negotiation.cc:189] Negotiated authn=SASL
0620 14:47:40.232584 (+ 831us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:47:40.232611 (+ 27us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:47:40.233854 (+ 1243us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:47:40.233859 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:47:40.235645 (+ 1786us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:47:40.235662 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:47:40.236490 (+ 828us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:47:40.236496 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:47:40.236696 (+ 200us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:47:40.237311 (+ 615us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 14:47:40.237334 (+ 23us) client_negotiation.cc:656] calling sasl_client_start()
0620 14:47:40.239504 (+ 2170us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 14:47:40.242337 (+ 2833us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 14:47:40.242344 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 14:47:40.242358 (+ 14us) client_negotiation.cc:760] calling sasl_client_step()
0620 14:47:40.242776 (+ 418us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 14:47:40.243091 (+ 315us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 14:47:40.243094 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 14:47:40.243097 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0620 14:47:40.243263 (+ 166us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 14:47:40.243781 (+ 518us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 14:47:40.243786 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 14:47:40.244106 (+ 320us) client_negotiation.cc:769] Sending connection context
0620 14:47:40.244330 (+ 224us) client_negotiation.cc:240] Negotiation successful
0620 14:47:40.244721 (+ 391us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":480,"thread_start_us":282,"threads_started":1}
I20260620 14:47:40.247813 3938 heartbeater.cc:347] Connected to a master server at 127.0.97.62:38725
I20260620 14:47:40.248154 3938 heartbeater.cc:464] Registering TS with master...
I20260620 14:47:40.248891 3938 heartbeater.cc:511] Master 127.0.97.62:38725 requested a full tablet report, sending...
I20260620 14:47:40.250236 3482 ts_manager.cc:194] Registered new tserver with Master: 01cc50c3dcbd4f399b3a9fc396c14211 (127.0.97.3:38439)
I20260620 14:47:40.251044 3482 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.97.3@KRBTEST.COM'} at 127.0.97.3:56099
I20260620 14:47:40.256203 3671 heartbeater.cc:503] Master 127.0.97.62:38725 was elected leader, sending a full tablet report...
I20260620 14:47:40.260947 388 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
Jun 20 14:47:40 dist-test-slave-q9j6 krb5kdc[3437](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781966860, etypes {rep=17 tkt=17 ses=17}, joe-interloper@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for joe-interloper@KRBTEST.COM:
Jun 20 14:47:40 dist-test-slave-q9j6 krb5kdc[3437](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781966860, etypes {rep=17 tkt=17 ses=17}, joe-interloper@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260620 14:47:40.309118 3673 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:47:40.295516 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50440 (local address 127.0.97.62:38725)
0620 14:47:40.295715 (+ 199us) server_negotiation.cc:206] Beginning negotiation
0620 14:47:40.295720 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 14:47:40.296136 (+ 416us) server_negotiation.cc:407] Connection header received
0620 14:47:40.296298 (+ 162us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:47:40.296302 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:47:40.296365 (+ 63us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:47:40.296471 (+ 106us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:47:40.297832 (+ 1361us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:47:40.298543 (+ 711us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:47:40.299820 (+ 1277us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:47:40.300242 (+ 422us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:47:40.300342 (+ 100us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:47:40.304267 (+ 3925us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 14:47:40.304272 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 14:47:40.304275 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 14:47:40.304311 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0620 14:47:40.306656 (+ 2345us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:47:40.307278 (+ 622us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:47:40.307281 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:47:40.307284 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:47:40.307396 (+ 112us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:47:40.307876 (+ 480us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:47:40.307881 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:47:40.307884 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:47:40.308261 (+ 377us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 14:47:40.308450 (+ 189us) server_negotiation.cc:1035] Waiting for connection context
0620 14:47:40.308718 (+ 268us) server_negotiation.cc:299] Negotiation successful
0620 14:47:40.308885 (+ 167us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":84}
W20260620 14:47:40.309695 3482 server_base.cc:1116] Unauthorized access attempt to method kudu.master.MasterService.ConnectToMaster from {username='joe-interloper', principal='joe-interloper@KRBTEST.COM'} at 127.0.0.1:50440
I20260620 14:47:40.311304 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskUci3DU/build/asan/bin/kudu pid 3543
Jun 20 14:47:40 dist-test-slave-q9j6 krb5kdc[3437](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781966860, etypes {rep=17 tkt=17 ses=17}, joe-interloper@KRBTEST.COM for kudu/127.0.97.1@KRBTEST.COM
I20260620 14:47:40.326957 3949 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:47:40.312569 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49554 (local address 127.0.97.1:35517)
0620 14:47:40.313181 (+ 612us) server_negotiation.cc:206] Beginning negotiation
0620 14:47:40.313189 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0620 14:47:40.313214 (+ 25us) server_negotiation.cc:407] Connection header received
0620 14:47:40.313267 (+ 53us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:47:40.313273 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:47:40.313487 (+ 214us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:47:40.313610 (+ 123us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:47:40.314718 (+ 1108us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:47:40.315863 (+ 1145us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:47:40.317023 (+ 1160us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:47:40.317584 (+ 561us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:47:40.317681 (+ 97us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:47:40.321584 (+ 3903us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 14:47:40.321595 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 14:47:40.321610 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 14:47:40.321649 (+ 39us) server_negotiation.cc:922] calling sasl_server_start()
0620 14:47:40.324283 (+ 2634us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:47:40.324961 (+ 678us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:47:40.324968 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:47:40.324971 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:47:40.325090 (+ 119us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:47:40.325458 (+ 368us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:47:40.325463 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:47:40.325465 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:47:40.325784 (+ 319us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 14:47:40.326067 (+ 283us) server_negotiation.cc:1035] Waiting for connection context
0620 14:47:40.326356 (+ 289us) server_negotiation.cc:299] Negotiation successful
0620 14:47:40.326616 (+ 260us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":450,"thread_start_us":321,"threads_started":1}
W20260620 14:47:40.328199 3666 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='joe-interloper', principal='joe-interloper@KRBTEST.COM'} at 127.0.0.1:49554
W20260620 14:47:40.329247 388 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskUci3DU/build/asan/bin/kudu pid 3543: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260620 14:47:40.329457 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskUci3DU/build/asan/bin/kudu with pid 3543
I20260620 14:47:40.345034 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskUci3DU/build/asan/bin/kudu pid 3677
Jun 20 14:47:40 dist-test-slave-q9j6 krb5kdc[3437](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781966860, etypes {rep=17 tkt=17 ses=17}, joe-interloper@KRBTEST.COM for kudu/127.0.97.2@KRBTEST.COM
I20260620 14:47:40.361094 3950 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:47:40.345933 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59882 (local address 127.0.97.2:35741)
0620 14:47:40.346495 (+ 562us) server_negotiation.cc:206] Beginning negotiation
0620 14:47:40.346503 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0620 14:47:40.346528 (+ 25us) server_negotiation.cc:407] Connection header received
0620 14:47:40.346588 (+ 60us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:47:40.346595 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:47:40.346819 (+ 224us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:47:40.346955 (+ 136us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:47:40.348165 (+ 1210us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:47:40.349137 (+ 972us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:47:40.350336 (+ 1199us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:47:40.350878 (+ 542us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:47:40.350975 (+ 97us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:47:40.355079 (+ 4104us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 14:47:40.355092 (+ 13us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 14:47:40.355108 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 14:47:40.355144 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0620 14:47:40.358130 (+ 2986us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:47:40.358838 (+ 708us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:47:40.358846 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:47:40.358848 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:47:40.358970 (+ 122us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:47:40.359355 (+ 385us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:47:40.359359 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:47:40.359361 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:47:40.359959 (+ 598us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 14:47:40.360191 (+ 232us) server_negotiation.cc:1035] Waiting for connection context
0620 14:47:40.360490 (+ 299us) server_negotiation.cc:299] Negotiation successful
0620 14:47:40.360753 (+ 263us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":419,"thread_start_us":289,"threads_started":1}
W20260620 14:47:40.362339 3800 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='joe-interloper', principal='joe-interloper@KRBTEST.COM'} at 127.0.0.1:59882
W20260620 14:47:40.363472 388 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskUci3DU/build/asan/bin/kudu pid 3677: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260620 14:47:40.363675 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskUci3DU/build/asan/bin/kudu with pid 3677
I20260620 14:47:40.379271 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskUci3DU/build/asan/bin/kudu pid 3810
Jun 20 14:47:40 dist-test-slave-q9j6 krb5kdc[3437](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781966860, etypes {rep=17 tkt=17 ses=17}, joe-interloper@KRBTEST.COM for kudu/127.0.97.3@KRBTEST.COM
I20260620 14:47:40.396010 3951 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:47:40.380333 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38504 (local address 127.0.97.3:38439)
0620 14:47:40.380922 (+ 589us) server_negotiation.cc:206] Beginning negotiation
0620 14:47:40.380929 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0620 14:47:40.380958 (+ 29us) server_negotiation.cc:407] Connection header received
0620 14:47:40.381019 (+ 61us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:47:40.381025 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:47:40.381235 (+ 210us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:47:40.381423 (+ 188us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:47:40.382790 (+ 1367us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:47:40.383919 (+ 1129us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:47:40.385184 (+ 1265us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:47:40.385764 (+ 580us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:47:40.385915 (+ 151us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:47:40.390394 (+ 4479us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 14:47:40.390407 (+ 13us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 14:47:40.390423 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 14:47:40.390462 (+ 39us) server_negotiation.cc:922] calling sasl_server_start()
0620 14:47:40.393149 (+ 2687us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:47:40.393837 (+ 688us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:47:40.393845 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:47:40.393849 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:47:40.393980 (+ 131us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:47:40.394355 (+ 375us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:47:40.394360 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:47:40.394364 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:47:40.394719 (+ 355us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 14:47:40.395037 (+ 318us) server_negotiation.cc:1035] Waiting for connection context
0620 14:47:40.395340 (+ 303us) server_negotiation.cc:299] Negotiation successful
0620 14:47:40.395636 (+ 296us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":448,"thread_start_us":299,"threads_started":1}
W20260620 14:47:40.397275 3933 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='joe-interloper', principal='joe-interloper@KRBTEST.COM'} at 127.0.0.1:38504
W20260620 14:47:40.398427 388 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskUci3DU/build/asan/bin/kudu pid 3810: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260620 14:47:40.398598 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskUci3DU/build/asan/bin/kudu with pid 3810
I20260620 14:47:40.414288 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskUci3DU/build/asan/bin/kudu pid 3450
I20260620 14:47:40.415546 3512 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 14:47:40.518211 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskUci3DU/build/asan/bin/kudu with pid 3450
2026-06-20T14:47:40Z chronyd exiting
[ OK ] SecurityITest.TestUnauthorizedClientKerberosCredentials (3594 ms)
[ RUN ] SecurityITest.TestAuthorizedSuperuser
Loading random data
Initializing database '/tmp/dist-test-taskUci3DU/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781966829091300-388-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 20 14:47:40 dist-test-slave-q9j6 krb5kdc[3955](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 20 14:47:40 dist-test-slave-q9j6 krb5kdc[3955](info): set up 2 sockets
Jun 20 14:47:40 dist-test-slave-q9j6 krb5kdc[3955](info): commencing operation
krb5kdc: starting...
W20260620 14:47:41.612279 388 mini_kdc.cc:121] Time spent starting KDC: real 1.052s user 0.002s sys 0.017s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskUci3DU/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781966829091300-388-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskUci3DU/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781966829091300-388-0/krb5kdc/test-user.keytab.
Jun 20 14:47:41 dist-test-slave-q9j6 krb5kdc[3955](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781966861, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-20T14:47:41Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T14:47:41Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.97.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskUci3DU/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781966829091300-388-0/krb5kdc/kudu_127.0.97.62.keytab.
Entry for principal kudu/127.0.97.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskUci3DU/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781966829091300-388-0/krb5kdc/kudu_127.0.97.62.keytab.
I20260620 14:47:41.809065 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskUci3DU/build/asan/bin/kudu
/tmp/dist-test-taskUci3DU/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskUci3DU/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781966829091300-388-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskUci3DU/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781966829091300-388-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskUci3DU/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781966829091300-388-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskUci3DU/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781966829091300-388-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:43437
--webserver_interface=127.0.97.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskUci3DU/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781966829091300-388-0/krb5kdc/kudu_127.0.97.62.keytab
--principal=kudu/127.0.97.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:44787
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.97.62:43437
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskUci3DU/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781966829091300-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskUci3DU/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781966829091300-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskUci3DU/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781966829091300-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 14:47:42.135250 3968 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 14:47:42.135567 3968 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 14:47:42.135757 3968 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 14:47:42.143177 3968 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 14:47:42.143297 3968 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 14:47:42.143388 3968 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 14:47:42.143504 3968 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 14:47:42.143565 3968 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 14:47:42.153479 3968 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:44787
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskUci3DU/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781966829091300-388-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskUci3DU/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781966829091300-388-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.97.62:43437
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskUci3DU/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781966829091300-388-0/krb5kdc/kudu_127.0.97.62.keytab
--principal=kudu/127.0.97.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:43437
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskUci3DU/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781966829091300-388-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.97.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskUci3DU/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781966829091300-388-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 13:41:20 UTC on ccycloud.quasar-vdagoi.root.comops.site
build id 4457462
ASAN enabled
I20260620 14:47:42.154839 3968 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 14:47:42.156208 3968 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 14:47:42.168068 3973 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 14:47:42.167999 3974 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 14:47:42.168025 3976 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 14:47:42.169257 3968 server_base.cc:1034] running on GCE node
I20260620 14:47:42.170030 3968 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 14:47:42.171419 3968 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 14:47:42.172624 3968 hybrid_clock.cc:648] HybridClock initialized: now 1781966862172572 us; error 78 us; skew 500 ppm
I20260620 14:47:42.173009 3968 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 14:47:42 dist-test-slave-q9j6 krb5kdc[3955](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781966862, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 14:47:42.179752 3968 init.cc:377] Logged in from keytab as kudu/127.0.97.62@KRBTEST.COM (short username kudu)
I20260620 14:47:42.181313 3968 webserver.cc:460] Webserver started at http://127.0.97.62:44663/ using document root <none> and password file <none>
I20260620 14:47:42.182006 3968 fs_manager.cc:359] Metadata directory not provided
I20260620 14:47:42.182111 3968 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 14:47:42.182415 3968 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 14:47:42.185571 3968 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskUci3DU/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781966829091300-388-0/minicluster-data/master-0/data/instance:
uuid: "68126fdeadbf45d3888e0a14f978c0a3"
format_stamp: "Formatted at 2026-06-20 14:47:42 on dist-test-slave-q9j6"
server_key: "84f86915f357b7dc43ccba2ffce22da8"
server_key_iv: "06ab7bb05f31c41dfd5c42d8cb7ca41f"
server_key_version: "encryptionkey@0"
I20260620 14:47:42.186504 3968 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskUci3DU/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781966829091300-388-0/minicluster-data/master-0/wal/instance:
uuid: "68126fdeadbf45d3888e0a14f978c0a3"
format_stamp: "Formatted at 2026-06-20 14:47:42 on dist-test-slave-q9j6"
server_key: "84f86915f357b7dc43ccba2ffce22da8"
server_key_iv: "06ab7bb05f31c41dfd5c42d8cb7ca41f"
server_key_version: "encryptionkey@0"
I20260620 14:47:42.192058 3968 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.007s sys 0.000s
I20260620 14:47:42.195577 3983 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 14:47:42.197125 3968 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260620 14:47:42.197336 3968 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskUci3DU/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781966829091300-388-0/minicluster-data/master-0/data,/tmp/dist-test-taskUci3DU/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781966829091300-388-0/minicluster-data/master-0/wal
uuid: "68126fdeadbf45d3888e0a14f978c0a3"
format_stamp: "Formatted at 2026-06-20 14:47:42 on dist-test-slave-q9j6"
server_key: "84f86915f357b7dc43ccba2ffce22da8"
server_key_iv: "06ab7bb05f31c41dfd5c42d8cb7ca41f"
server_key_version: "encryptionkey@0"
I20260620 14:47:42.197556 3968 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskUci3DU/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781966829091300-388-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskUci3DU/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781966829091300-388-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskUci3DU/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781966829091300-388-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 14:47:42.220999 3968 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 14:47:42.225330 3968 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 14:47:42.225632 3968 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 14:47:42.243353 3968 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.62:43437
I20260620 14:47:42.243376 4034 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.62:43437 every 8 connection(s)
I20260620 14:47:42.244804 3968 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskUci3DU/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781966829091300-388-0/minicluster-data/master-0/data/info.pb
I20260620 14:47:42.248406 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskUci3DU/build/asan/bin/kudu as pid 3968
I20260620 14:47:42.248627 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskUci3DU/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781966829091300-388-0/minicluster-data/master-0/wal/instance
I20260620 14:47:42.248663 4035 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 14:47:42.249269 388 external_mini_cluster.cc:1428] Setting key aed2433fd97d9df669e69005d6c80782
I20260620 14:47:42.263325 4035 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 68126fdeadbf45d3888e0a14f978c0a3: Bootstrap starting.
Jun 20 14:47:42 dist-test-slave-q9j6 krb5kdc[3955](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781966861, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260620 14:47:42.267218 4035 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 68126fdeadbf45d3888e0a14f978c0a3: Neither blocks nor log segments found. Creating new log.
I20260620 14:47:42.268589 4035 log.cc:824] T 00000000000000000000000000000000 P 68126fdeadbf45d3888e0a14f978c0a3: Log is configured to *not* fsync() on all Append() calls
I20260620 14:47:42.272020 4035 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 68126fdeadbf45d3888e0a14f978c0a3: No bootstrap required, opened a new log
I20260620 14:47:42.273559 4038 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:47:42.251273 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56902 (local address 127.0.97.62:43437)
0620 14:47:42.252116 (+ 843us) server_negotiation.cc:206] Beginning negotiation
0620 14:47:42.252132 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0620 14:47:42.252189 (+ 57us) server_negotiation.cc:407] Connection header received
0620 14:47:42.253997 (+ 1808us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:47:42.254034 (+ 37us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:47:42.254579 (+ 545us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:47:42.255066 (+ 487us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:47:42.256559 (+ 1493us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:47:42.258456 (+ 1897us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:47:42.259802 (+ 1346us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:47:42.260924 (+ 1122us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:47:42.261121 (+ 197us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:47:42.264989 (+ 3868us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 14:47:42.265001 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 14:47:42.265023 (+ 22us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 14:47:42.265084 (+ 61us) server_negotiation.cc:922] calling sasl_server_start()
0620 14:47:42.268941 (+ 3857us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:47:42.269807 (+ 866us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:47:42.269813 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:47:42.269819 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:47:42.270035 (+ 216us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:47:42.270393 (+ 358us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:47:42.270397 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:47:42.270400 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:47:42.271073 (+ 673us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 14:47:42.271346 (+ 273us) server_negotiation.cc:1035] Waiting for connection context
0620 14:47:42.271762 (+ 416us) server_negotiation.cc:299] Negotiation successful
0620 14:47:42.272193 (+ 431us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":591,"thread_start_us":360,"threads_started":1}
I20260620 14:47:42.280951 4035 raft_consensus.cc:348] T 00000000000000000000000000000000 P 68126fdeadbf45d3888e0a14f978c0a3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "68126fdeadbf45d3888e0a14f978c0a3" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 43437 } }
I20260620 14:47:42.281320 4035 raft_consensus.cc:374] T 00000000000000000000000000000000 P 68126fdeadbf45d3888e0a14f978c0a3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 14:47:42.281388 4035 raft_consensus.cc:729] T 00000000000000000000000000000000 P 68126fdeadbf45d3888e0a14f978c0a3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 68126fdeadbf45d3888e0a14f978c0a3, State: Initialized, Role: FOLLOWER
I20260620 14:47:42.281955 4035 consensus_queue.cc:260] T 00000000000000000000000000000000 P 68126fdeadbf45d3888e0a14f978c0a3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "68126fdeadbf45d3888e0a14f978c0a3" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 43437 } }
I20260620 14:47:42.282155 4035 raft_consensus.cc:388] T 00000000000000000000000000000000 P 68126fdeadbf45d3888e0a14f978c0a3 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 14:47:42.282251 4035 raft_consensus.cc:482] T 00000000000000000000000000000000 P 68126fdeadbf45d3888e0a14f978c0a3 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 14:47:42.282384 4035 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 68126fdeadbf45d3888e0a14f978c0a3 [term 0 FOLLOWER]: Advancing to term 1
I20260620 14:47:42.284577 4035 raft_consensus.cc:504] T 00000000000000000000000000000000 P 68126fdeadbf45d3888e0a14f978c0a3 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "68126fdeadbf45d3888e0a14f978c0a3" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 43437 } }
I20260620 14:47:42.285041 4035 leader_election.cc:302] T 00000000000000000000000000000000 P 68126fdeadbf45d3888e0a14f978c0a3 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 68126fdeadbf45d3888e0a14f978c0a3; no voters:
I20260620 14:47:42.285684 4035 leader_election.cc:288] T 00000000000000000000000000000000 P 68126fdeadbf45d3888e0a14f978c0a3 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260620 14:47:42.285815 4040 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 68126fdeadbf45d3888e0a14f978c0a3 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 14:47:42.286793 4040 raft_consensus.cc:686] T 00000000000000000000000000000000 P 68126fdeadbf45d3888e0a14f978c0a3 [term 1 LEADER]: Becoming Leader. State: Replica: 68126fdeadbf45d3888e0a14f978c0a3, State: Running, Role: LEADER
I20260620 14:47:42.287369 4040 consensus_queue.cc:237] T 00000000000000000000000000000000 P 68126fdeadbf45d3888e0a14f978c0a3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "68126fdeadbf45d3888e0a14f978c0a3" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 43437 } }
I20260620 14:47:42.287918 4035 sys_catalog.cc:564] T 00000000000000000000000000000000 P 68126fdeadbf45d3888e0a14f978c0a3 [sys.catalog]: configured and running, proceeding with master startup.
I20260620 14:47:42.290663 4041 sys_catalog.cc:455] T 00000000000000000000000000000000 P 68126fdeadbf45d3888e0a14f978c0a3 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "68126fdeadbf45d3888e0a14f978c0a3" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "68126fdeadbf45d3888e0a14f978c0a3" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 43437 } } }
I20260620 14:47:42.291039 4041 sys_catalog.cc:458] T 00000000000000000000000000000000 P 68126fdeadbf45d3888e0a14f978c0a3 [sys.catalog]: This master's current role is: LEADER
I20260620 14:47:42.290768 4042 sys_catalog.cc:455] T 00000000000000000000000000000000 P 68126fdeadbf45d3888e0a14f978c0a3 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 68126fdeadbf45d3888e0a14f978c0a3. Latest consensus state: current_term: 1 leader_uuid: "68126fdeadbf45d3888e0a14f978c0a3" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "68126fdeadbf45d3888e0a14f978c0a3" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 43437 } } }
I20260620 14:47:42.291152 4042 sys_catalog.cc:458] T 00000000000000000000000000000000 P 68126fdeadbf45d3888e0a14f978c0a3 [sys.catalog]: This master's current role is: LEADER
I20260620 14:47:42.293633 4048 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 14:47:42.299978 4048 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 14:47:42.309180 4048 catalog_manager.cc:1347] Generated new cluster ID: 01c3b19c52cb4551a91675201f18b86b
I20260620 14:47:42.309317 4048 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 14:47:42.324452 4048 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 14:47:42.325912 4048 catalog_manager.cc:1504] Loading token signing keys...
I20260620 14:47:42.332593 4048 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 68126fdeadbf45d3888e0a14f978c0a3: Generated new TSK 0
I20260620 14:47:42.333477 4048 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.0.97.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskUci3DU/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781966829091300-388-0/krb5kdc/kudu_127.0.97.1.keytab.
Entry for principal kudu/127.0.97.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskUci3DU/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781966829091300-388-0/krb5kdc/kudu_127.0.97.1.keytab.
I20260620 14:47:42.384665 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskUci3DU/build/asan/bin/kudu
/tmp/dist-test-taskUci3DU/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskUci3DU/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781966829091300-388-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskUci3DU/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781966829091300-388-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskUci3DU/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781966829091300-388-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskUci3DU/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781966829091300-388-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.1:0
--local_ip_for_outbound_sockets=127.0.97.1
--webserver_interface=127.0.97.1
--webserver_port=0
--tserver_master_addrs=127.0.97.62:43437
--keytab_file=/tmp/dist-test-taskUci3DU/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781966829091300-388-0/krb5kdc/kudu_127.0.97.1.keytab
--principal=kudu/127.0.97.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:44787
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskUci3DU/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781966829091300-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskUci3DU/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781966829091300-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskUci3DU/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781966829091300-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 14:47:42.718308 4061 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 14:47:42.718623 4061 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 14:47:42.718791 4061 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 14:47:42.726369 4061 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 14:47:42.726528 4061 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 14:47:42.726725 4061 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.1
I20260620 14:47:42.736742 4061 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:44787
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskUci3DU/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781966829091300-388-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskUci3DU/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781966829091300-388-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskUci3DU/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781966829091300-388-0/krb5kdc/kudu_127.0.97.1.keytab
--principal=kudu/127.0.97.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskUci3DU/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781966829091300-388-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.97.1
--webserver_port=0
--tserver_master_addrs=127.0.97.62:43437
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.1
--log_dir=/tmp/dist-test-taskUci3DU/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781966829091300-388-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 13:41:20 UTC on ccycloud.quasar-vdagoi.root.comops.site
build id 4457462
ASAN enabled
I20260620 14:47:42.738098 4061 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 14:47:42.739439 4061 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 14:47:42.752895 4069 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 14:47:42.753011 4067 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 14:47:42.753011 4066 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 14:47:42.753625 4061 server_base.cc:1034] running on GCE node
I20260620 14:47:42.754318 4061 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 14:47:42.755251 4061 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 14:47:42.756443 4061 hybrid_clock.cc:648] HybridClock initialized: now 1781966862756397 us; error 48 us; skew 500 ppm
I20260620 14:47:42.756773 4061 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 14:47:42 dist-test-slave-q9j6 krb5kdc[3955](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781966862, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 14:47:42.759958 4061 init.cc:377] Logged in from keytab as kudu/127.0.97.1@KRBTEST.COM (short username kudu)
I20260620 14:47:42.761471 4061 webserver.cc:460] Webserver started at http://127.0.97.1:45811/ using document root <none> and password file <none>
I20260620 14:47:42.762178 4061 fs_manager.cc:359] Metadata directory not provided
I20260620 14:47:42.762326 4061 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 14:47:42.762749 4061 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 14:47:42.765770 4061 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskUci3DU/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781966829091300-388-0/minicluster-data/ts-0/data/instance:
uuid: "317251b6e73846189cb7ce2a0d6042ed"
format_stamp: "Formatted at 2026-06-20 14:47:42 on dist-test-slave-q9j6"
server_key: "a3717880bd8a738a080b0f05d3c15a30"
server_key_iv: "db11c5cc9ff0aadc9c3a707046cf69b9"
server_key_version: "encryptionkey@0"
I20260620 14:47:42.766594 4061 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskUci3DU/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781966829091300-388-0/minicluster-data/ts-0/wal/instance:
uuid: "317251b6e73846189cb7ce2a0d6042ed"
format_stamp: "Formatted at 2026-06-20 14:47:42 on dist-test-slave-q9j6"
server_key: "a3717880bd8a738a080b0f05d3c15a30"
server_key_iv: "db11c5cc9ff0aadc9c3a707046cf69b9"
server_key_version: "encryptionkey@0"
I20260620 14:47:42.772022 4061 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.001s
I20260620 14:47:42.775975 4076 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 14:47:42.777449 4061 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.001s
I20260620 14:47:42.777657 4061 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskUci3DU/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781966829091300-388-0/minicluster-data/ts-0/data,/tmp/dist-test-taskUci3DU/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781966829091300-388-0/minicluster-data/ts-0/wal
uuid: "317251b6e73846189cb7ce2a0d6042ed"
format_stamp: "Formatted at 2026-06-20 14:47:42 on dist-test-slave-q9j6"
server_key: "a3717880bd8a738a080b0f05d3c15a30"
server_key_iv: "db11c5cc9ff0aadc9c3a707046cf69b9"
server_key_version: "encryptionkey@0"
I20260620 14:47:42.777877 4061 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskUci3DU/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781966829091300-388-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskUci3DU/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781966829091300-388-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskUci3DU/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781966829091300-388-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 14:47:42.802628 4061 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 14:47:42.806900 4061 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 14:47:42.807216 4061 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 14:47:42.808298 4061 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 14:47:42.809659 4061 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 14:47:42.809770 4061 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 14:47:42.809926 4061 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 14:47:42.809998 4061 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 14:47:42.844192 4061 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.1:44803
I20260620 14:47:42.844350 4188 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.1:44803 every 8 connection(s)
I20260620 14:47:42.845535 4061 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskUci3DU/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781966829091300-388-0/minicluster-data/ts-0/data/info.pb
I20260620 14:47:42.854816 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskUci3DU/build/asan/bin/kudu as pid 4061
I20260620 14:47:42.855028 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskUci3DU/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781966829091300-388-0/minicluster-data/ts-0/wal/instance
I20260620 14:47:42.855724 388 external_mini_cluster.cc:1428] Setting key 895b52aa97a059a02221252ff9eb701a
Jun 20 14:47:42 dist-test-slave-q9j6 krb5kdc[3955](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781966862, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.1@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260620 14:47:42.866794 4191 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:47:42.848278 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.1:47901 (local address 127.0.97.62:43437)
0620 14:47:42.848735 (+ 457us) server_negotiation.cc:206] Beginning negotiation
0620 14:47:42.848740 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 14:47:42.849891 (+ 1151us) server_negotiation.cc:407] Connection header received
0620 14:47:42.851782 (+ 1891us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:47:42.851788 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:47:42.851892 (+ 104us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:47:42.852031 (+ 139us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:47:42.853598 (+ 1567us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:47:42.854316 (+ 718us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:47:42.856122 (+ 1806us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:47:42.856703 (+ 581us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:47:42.856850 (+ 147us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:47:42.861011 (+ 4161us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 14:47:42.861016 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 14:47:42.861019 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 14:47:42.861052 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0620 14:47:42.863672 (+ 2620us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:47:42.864443 (+ 771us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:47:42.864455 (+ 12us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:47:42.864458 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:47:42.864580 (+ 122us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:47:42.865024 (+ 444us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:47:42.865031 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:47:42.865038 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:47:42.865294 (+ 256us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 14:47:42.865449 (+ 155us) server_negotiation.cc:1035] Waiting for connection context
0620 14:47:42.866198 (+ 749us) server_negotiation.cc:299] Negotiation successful
0620 14:47:42.866426 (+ 228us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":354,"thread_start_us":230,"threads_started":1}
I20260620 14:47:42.867895 4192 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:47:42.848961 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:43437 (local address 127.0.97.1:47901)
0620 14:47:42.849712 (+ 751us) negotiation.cc:107] Waiting for socket to connect
0620 14:47:42.849752 (+ 40us) client_negotiation.cc:174] Beginning negotiation
0620 14:47:42.851440 (+ 1688us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 14:47:42.852279 (+ 839us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 14:47:42.852288 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 14:47:42.852840 (+ 552us) client_negotiation.cc:189] Negotiated authn=SASL
0620 14:47:42.853413 (+ 573us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:47:42.853429 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:47:42.854493 (+ 1064us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:47:42.854498 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:47:42.855980 (+ 1482us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:47:42.855994 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:47:42.856954 (+ 960us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:47:42.856959 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:47:42.857175 (+ 216us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:47:42.857974 (+ 799us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 14:47:42.858005 (+ 31us) client_negotiation.cc:656] calling sasl_client_start()
0620 14:47:42.860767 (+ 2762us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 14:47:42.863819 (+ 3052us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 14:47:42.863829 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 14:47:42.863848 (+ 19us) client_negotiation.cc:760] calling sasl_client_step()
0620 14:47:42.864305 (+ 457us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 14:47:42.864693 (+ 388us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 14:47:42.864698 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 14:47:42.864702 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0620 14:47:42.864900 (+ 198us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 14:47:42.865427 (+ 527us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 14:47:42.865436 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 14:47:42.865796 (+ 360us) client_negotiation.cc:769] Sending connection context
0620 14:47:42.866090 (+ 294us) client_negotiation.cc:240] Negotiation successful
0620 14:47:42.866650 (+ 560us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":510,"thread_start_us":296,"threads_started":1}
I20260620 14:47:42.869540 4189 heartbeater.cc:347] Connected to a master server at 127.0.97.62:43437
I20260620 14:47:42.869907 4189 heartbeater.cc:464] Registering TS with master...
I20260620 14:47:42.870863 4189 heartbeater.cc:511] Master 127.0.97.62:43437 requested a full tablet report, sending...
I20260620 14:47:42.873268 4000 ts_manager.cc:194] Registered new tserver with Master: 317251b6e73846189cb7ce2a0d6042ed (127.0.97.1:44803)
WARNING: no policy specified for kudu/127.0.97.2@KRBTEST.COM; defaulting to no policy
I20260620 14:47:42.874753 4000 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.97.1@KRBTEST.COM'} at 127.0.97.1:47901
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskUci3DU/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781966829091300-388-0/krb5kdc/kudu_127.0.97.2.keytab.
Entry for principal kudu/127.0.97.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskUci3DU/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781966829091300-388-0/krb5kdc/kudu_127.0.97.2.keytab.
I20260620 14:47:42.898689 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskUci3DU/build/asan/bin/kudu
/tmp/dist-test-taskUci3DU/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskUci3DU/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781966829091300-388-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskUci3DU/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781966829091300-388-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskUci3DU/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781966829091300-388-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskUci3DU/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781966829091300-388-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.2:0
--local_ip_for_outbound_sockets=127.0.97.2
--webserver_interface=127.0.97.2
--webserver_port=0
--tserver_master_addrs=127.0.97.62:43437
--keytab_file=/tmp/dist-test-taskUci3DU/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781966829091300-388-0/krb5kdc/kudu_127.0.97.2.keytab
--principal=kudu/127.0.97.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:44787
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskUci3DU/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781966829091300-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskUci3DU/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781966829091300-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskUci3DU/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781966829091300-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 14:47:43.227296 4195 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 14:47:43.227691 4195 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 14:47:43.227805 4195 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 14:47:43.235183 4195 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 14:47:43.235303 4195 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 14:47:43.235419 4195 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.2
I20260620 14:47:43.245599 4195 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:44787
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskUci3DU/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781966829091300-388-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskUci3DU/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781966829091300-388-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskUci3DU/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781966829091300-388-0/krb5kdc/kudu_127.0.97.2.keytab
--principal=kudu/127.0.97.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskUci3DU/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781966829091300-388-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.97.2
--webserver_port=0
--tserver_master_addrs=127.0.97.62:43437
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.2
--log_dir=/tmp/dist-test-taskUci3DU/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781966829091300-388-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 13:41:20 UTC on ccycloud.quasar-vdagoi.root.comops.site
build id 4457462
ASAN enabled
I20260620 14:47:43.247093 4195 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 14:47:43.248457 4195 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 14:47:43.262668 4200 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 14:47:43.262688 4203 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 14:47:43.262621 4201 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 14:47:43.263533 4195 server_base.cc:1034] running on GCE node
I20260620 14:47:43.264213 4195 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 14:47:43.265177 4195 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 14:47:43.266386 4195 hybrid_clock.cc:648] HybridClock initialized: now 1781966863266311 us; error 68 us; skew 500 ppm
I20260620 14:47:43.266783 4195 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 14:47:43 dist-test-slave-q9j6 krb5kdc[3955](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781966863, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 14:47:43.270124 4195 init.cc:377] Logged in from keytab as kudu/127.0.97.2@KRBTEST.COM (short username kudu)
I20260620 14:47:43.271584 4195 webserver.cc:460] Webserver started at http://127.0.97.2:34631/ using document root <none> and password file <none>
I20260620 14:47:43.272408 4195 fs_manager.cc:359] Metadata directory not provided
I20260620 14:47:43.272562 4195 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 14:47:43.272883 4195 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 14:47:43.276090 4195 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskUci3DU/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781966829091300-388-0/minicluster-data/ts-1/data/instance:
uuid: "fe9c56d2520b41d7b37b94d287c61ef3"
format_stamp: "Formatted at 2026-06-20 14:47:43 on dist-test-slave-q9j6"
server_key: "32a4f6520663c9f9ae7992c45ce77175"
server_key_iv: "8bf848c70b46a04bbfda82a4d1f0fb4c"
server_key_version: "encryptionkey@0"
I20260620 14:47:43.276902 4195 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskUci3DU/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781966829091300-388-0/minicluster-data/ts-1/wal/instance:
uuid: "fe9c56d2520b41d7b37b94d287c61ef3"
format_stamp: "Formatted at 2026-06-20 14:47:43 on dist-test-slave-q9j6"
server_key: "32a4f6520663c9f9ae7992c45ce77175"
server_key_iv: "8bf848c70b46a04bbfda82a4d1f0fb4c"
server_key_version: "encryptionkey@0"
I20260620 14:47:43.282378 4195 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260620 14:47:43.285928 4210 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 14:47:43.287319 4195 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.000s sys 0.002s
I20260620 14:47:43.287459 4195 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskUci3DU/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781966829091300-388-0/minicluster-data/ts-1/data,/tmp/dist-test-taskUci3DU/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781966829091300-388-0/minicluster-data/ts-1/wal
uuid: "fe9c56d2520b41d7b37b94d287c61ef3"
format_stamp: "Formatted at 2026-06-20 14:47:43 on dist-test-slave-q9j6"
server_key: "32a4f6520663c9f9ae7992c45ce77175"
server_key_iv: "8bf848c70b46a04bbfda82a4d1f0fb4c"
server_key_version: "encryptionkey@0"
I20260620 14:47:43.287703 4195 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskUci3DU/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781966829091300-388-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskUci3DU/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781966829091300-388-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskUci3DU/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781966829091300-388-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 14:47:43.307494 4195 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 14:47:43.311427 4195 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 14:47:43.311762 4195 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 14:47:43.312723 4195 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 14:47:43.314059 4195 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 14:47:43.314131 4195 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 14:47:43.314221 4195 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 14:47:43.314265 4195 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 14:47:43.347024 4195 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.2:38457
I20260620 14:47:43.347185 4322 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.2:38457 every 8 connection(s)
I20260620 14:47:43.348417 4195 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskUci3DU/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781966829091300-388-0/minicluster-data/ts-1/data/info.pb
I20260620 14:47:43.358331 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskUci3DU/build/asan/bin/kudu as pid 4195
I20260620 14:47:43.358516 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskUci3DU/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781966829091300-388-0/minicluster-data/ts-1/wal/instance
I20260620 14:47:43.359026 388 external_mini_cluster.cc:1428] Setting key 188edc782c49e3d38453b8ee76cd5b5f
Jun 20 14:47:43 dist-test-slave-q9j6 krb5kdc[3955](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781966863, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.2@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260620 14:47:43.369568 4191 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:47:43.350963 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.2:50633 (local address 127.0.97.62:43437)
0620 14:47:43.351129 (+ 166us) server_negotiation.cc:206] Beginning negotiation
0620 14:47:43.351133 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 14:47:43.352336 (+ 1203us) server_negotiation.cc:407] Connection header received
0620 14:47:43.354103 (+ 1767us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:47:43.354108 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:47:43.354201 (+ 93us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:47:43.354307 (+ 106us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:47:43.355859 (+ 1552us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:47:43.356556 (+ 697us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:47:43.358157 (+ 1601us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:47:43.358691 (+ 534us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:47:43.358795 (+ 104us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:47:43.362830 (+ 4035us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 14:47:43.362835 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 14:47:43.362838 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 14:47:43.362870 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0620 14:47:43.366145 (+ 3275us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:47:43.367129 (+ 984us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:47:43.367133 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:47:43.367135 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:47:43.367256 (+ 121us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:47:43.367743 (+ 487us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:47:43.367751 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:47:43.367757 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:47:43.368148 (+ 391us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 14:47:43.368297 (+ 149us) server_negotiation.cc:1035] Waiting for connection context
0620 14:47:43.368999 (+ 702us) server_negotiation.cc:299] Negotiation successful
0620 14:47:43.369227 (+ 228us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":53}
I20260620 14:47:43.370605 4325 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:47:43.351402 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:43437 (local address 127.0.97.2:50633)
0620 14:47:43.352178 (+ 776us) negotiation.cc:107] Waiting for socket to connect
0620 14:47:43.352216 (+ 38us) client_negotiation.cc:174] Beginning negotiation
0620 14:47:43.353809 (+ 1593us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 14:47:43.354548 (+ 739us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 14:47:43.354557 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 14:47:43.355072 (+ 515us) client_negotiation.cc:189] Negotiated authn=SASL
0620 14:47:43.355694 (+ 622us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:47:43.355713 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:47:43.356699 (+ 986us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:47:43.356704 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:47:43.357985 (+ 1281us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:47:43.357999 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:47:43.358918 (+ 919us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:47:43.358925 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:47:43.359241 (+ 316us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:47:43.360155 (+ 914us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 14:47:43.360181 (+ 26us) client_negotiation.cc:656] calling sasl_client_start()
0620 14:47:43.362600 (+ 2419us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 14:47:43.366296 (+ 3696us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 14:47:43.366308 (+ 12us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 14:47:43.366330 (+ 22us) client_negotiation.cc:760] calling sasl_client_step()
0620 14:47:43.367006 (+ 676us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 14:47:43.367349 (+ 343us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 14:47:43.367353 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 14:47:43.367355 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 14:47:43.367571 (+ 216us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 14:47:43.368268 (+ 697us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 14:47:43.368274 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 14:47:43.368622 (+ 348us) client_negotiation.cc:769] Sending connection context
0620 14:47:43.368894 (+ 272us) client_negotiation.cc:240] Negotiation successful
0620 14:47:43.369443 (+ 549us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":549,"thread_start_us":345,"threads_started":1}
I20260620 14:47:43.372241 4323 heartbeater.cc:347] Connected to a master server at 127.0.97.62:43437
I20260620 14:47:43.372552 4323 heartbeater.cc:464] Registering TS with master...
I20260620 14:47:43.373288 4323 heartbeater.cc:511] Master 127.0.97.62:43437 requested a full tablet report, sending...
I20260620 14:47:43.374796 4000 ts_manager.cc:194] Registered new tserver with Master: fe9c56d2520b41d7b37b94d287c61ef3 (127.0.97.2:38457)
I20260620 14:47:43.375674 4000 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.97.2@KRBTEST.COM'} at 127.0.97.2:50633
WARNING: no policy specified for kudu/127.0.97.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskUci3DU/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781966829091300-388-0/krb5kdc/kudu_127.0.97.3.keytab.
Entry for principal kudu/127.0.97.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskUci3DU/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781966829091300-388-0/krb5kdc/kudu_127.0.97.3.keytab.
I20260620 14:47:43.403214 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskUci3DU/build/asan/bin/kudu
/tmp/dist-test-taskUci3DU/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskUci3DU/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781966829091300-388-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskUci3DU/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781966829091300-388-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskUci3DU/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781966829091300-388-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskUci3DU/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781966829091300-388-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.3:0
--local_ip_for_outbound_sockets=127.0.97.3
--webserver_interface=127.0.97.3
--webserver_port=0
--tserver_master_addrs=127.0.97.62:43437
--keytab_file=/tmp/dist-test-taskUci3DU/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781966829091300-388-0/krb5kdc/kudu_127.0.97.3.keytab
--principal=kudu/127.0.97.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:44787
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskUci3DU/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781966829091300-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskUci3DU/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781966829091300-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskUci3DU/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781966829091300-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 14:47:43.726931 4328 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 14:47:43.727218 4328 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 14:47:43.727311 4328 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 14:47:43.734786 4328 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 14:47:43.734900 4328 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 14:47:43.735014 4328 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.3
I20260620 14:47:43.744663 4328 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:44787
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskUci3DU/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781966829091300-388-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskUci3DU/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781966829091300-388-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskUci3DU/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781966829091300-388-0/krb5kdc/kudu_127.0.97.3.keytab
--principal=kudu/127.0.97.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskUci3DU/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781966829091300-388-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.97.3
--webserver_port=0
--tserver_master_addrs=127.0.97.62:43437
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.3
--log_dir=/tmp/dist-test-taskUci3DU/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781966829091300-388-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 13:41:20 UTC on ccycloud.quasar-vdagoi.root.comops.site
build id 4457462
ASAN enabled
I20260620 14:47:43.745919 4328 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 14:47:43.747161 4328 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 14:47:43.760488 4336 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 14:47:43.760736 4334 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 14:47:43.760429 4333 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 14:47:43.761204 4328 server_base.cc:1034] running on GCE node
I20260620 14:47:43.761904 4328 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 14:47:43.762892 4328 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 14:47:43.764088 4328 hybrid_clock.cc:648] HybridClock initialized: now 1781966863764041 us; error 38 us; skew 500 ppm
I20260620 14:47:43.764417 4328 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 14:47:43 dist-test-slave-q9j6 krb5kdc[3955](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781966863, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 14:47:43.767647 4328 init.cc:377] Logged in from keytab as kudu/127.0.97.3@KRBTEST.COM (short username kudu)
I20260620 14:47:43.769058 4328 webserver.cc:460] Webserver started at http://127.0.97.3:45997/ using document root <none> and password file <none>
I20260620 14:47:43.769744 4328 fs_manager.cc:359] Metadata directory not provided
I20260620 14:47:43.769879 4328 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 14:47:43.770203 4328 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 14:47:43.773226 4328 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskUci3DU/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781966829091300-388-0/minicluster-data/ts-2/data/instance:
uuid: "9a471ae88ebf43aeb2e91ecc41a74ff9"
format_stamp: "Formatted at 2026-06-20 14:47:43 on dist-test-slave-q9j6"
server_key: "b7a7353a1f309a8c92438a51aaaa1081"
server_key_iv: "7108ec0ca728d6bb6d9e2375c23b210b"
server_key_version: "encryptionkey@0"
I20260620 14:47:43.774025 4328 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskUci3DU/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781966829091300-388-0/minicluster-data/ts-2/wal/instance:
uuid: "9a471ae88ebf43aeb2e91ecc41a74ff9"
format_stamp: "Formatted at 2026-06-20 14:47:43 on dist-test-slave-q9j6"
server_key: "b7a7353a1f309a8c92438a51aaaa1081"
server_key_iv: "7108ec0ca728d6bb6d9e2375c23b210b"
server_key_version: "encryptionkey@0"
I20260620 14:47:43.779312 4328 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.000s
I20260620 14:47:43.783113 4343 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 14:47:43.784636 4328 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260620 14:47:43.784812 4328 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskUci3DU/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781966829091300-388-0/minicluster-data/ts-2/data,/tmp/dist-test-taskUci3DU/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781966829091300-388-0/minicluster-data/ts-2/wal
uuid: "9a471ae88ebf43aeb2e91ecc41a74ff9"
format_stamp: "Formatted at 2026-06-20 14:47:43 on dist-test-slave-q9j6"
server_key: "b7a7353a1f309a8c92438a51aaaa1081"
server_key_iv: "7108ec0ca728d6bb6d9e2375c23b210b"
server_key_version: "encryptionkey@0"
I20260620 14:47:43.785027 4328 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskUci3DU/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781966829091300-388-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskUci3DU/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781966829091300-388-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskUci3DU/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781966829091300-388-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 14:47:43.797221 4328 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 14:47:43.801633 4328 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 14:47:43.801936 4328 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 14:47:43.802907 4328 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 14:47:43.804298 4328 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 14:47:43.804415 4328 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 14:47:43.804563 4328 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 14:47:43.804637 4328 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 14:47:43.838074 4328 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.3:45373
I20260620 14:47:43.838124 4455 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.3:45373 every 8 connection(s)
I20260620 14:47:43.839445 4328 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskUci3DU/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781966829091300-388-0/minicluster-data/ts-2/data/info.pb
I20260620 14:47:43.842214 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskUci3DU/build/asan/bin/kudu as pid 4328
I20260620 14:47:43.842393 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskUci3DU/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781966829091300-388-0/minicluster-data/ts-2/wal/instance
I20260620 14:47:43.842912 388 external_mini_cluster.cc:1428] Setting key 9d8d1f10351ab0a6b869a07b80803aab
Jun 20 14:47:43 dist-test-slave-q9j6 krb5kdc[3955](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781966863, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.3@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260620 14:47:43.861354 4191 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:47:43.842126 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.3:48025 (local address 127.0.97.62:43437)
0620 14:47:43.842327 (+ 201us) server_negotiation.cc:206] Beginning negotiation
0620 14:47:43.842333 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 14:47:43.843977 (+ 1644us) server_negotiation.cc:407] Connection header received
0620 14:47:43.846608 (+ 2631us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:47:43.846612 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:47:43.846679 (+ 67us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:47:43.846759 (+ 80us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:47:43.848867 (+ 2108us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:47:43.849636 (+ 769us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:47:43.851271 (+ 1635us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:47:43.851734 (+ 463us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:47:43.851833 (+ 99us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:47:43.855206 (+ 3373us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 14:47:43.855211 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 14:47:43.855214 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 14:47:43.855251 (+ 37us) server_negotiation.cc:922] calling sasl_server_start()
0620 14:47:43.858309 (+ 3058us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:47:43.859214 (+ 905us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:47:43.859218 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:47:43.859220 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:47:43.859347 (+ 127us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:47:43.859825 (+ 478us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:47:43.859830 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:47:43.859834 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:47:43.860125 (+ 291us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 14:47:43.860235 (+ 110us) server_negotiation.cc:1035] Waiting for connection context
0620 14:47:43.860943 (+ 708us) server_negotiation.cc:299] Negotiation successful
0620 14:47:43.861118 (+ 175us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":58}
I20260620 14:47:43.862331 4458 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:47:43.842824 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:43437 (local address 127.0.97.3:48025)
0620 14:47:43.843815 (+ 991us) negotiation.cc:107] Waiting for socket to connect
0620 14:47:43.843863 (+ 48us) client_negotiation.cc:174] Beginning negotiation
0620 14:47:43.846291 (+ 2428us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 14:47:43.847052 (+ 761us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 14:47:43.847065 (+ 13us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 14:47:43.847904 (+ 839us) client_negotiation.cc:189] Negotiated authn=SASL
0620 14:47:43.848678 (+ 774us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:47:43.848705 (+ 27us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:47:43.849800 (+ 1095us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:47:43.849807 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:47:43.851123 (+ 1316us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:47:43.851140 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:47:43.851927 (+ 787us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:47:43.851932 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:47:43.852141 (+ 209us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:47:43.852770 (+ 629us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 14:47:43.852793 (+ 23us) client_negotiation.cc:656] calling sasl_client_start()
0620 14:47:43.854973 (+ 2180us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 14:47:43.858493 (+ 3520us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 14:47:43.858502 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 14:47:43.858523 (+ 21us) client_negotiation.cc:760] calling sasl_client_step()
0620 14:47:43.859105 (+ 582us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 14:47:43.859454 (+ 349us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 14:47:43.859458 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 14:47:43.859460 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 14:47:43.859720 (+ 260us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 14:47:43.860234 (+ 514us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 14:47:43.860241 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 14:47:43.860589 (+ 348us) client_negotiation.cc:769] Sending connection context
0620 14:47:43.860832 (+ 243us) client_negotiation.cc:240] Negotiation successful
0620 14:47:43.861228 (+ 396us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":648,"thread_start_us":403,"threads_started":1}
I20260620 14:47:43.864073 4456 heartbeater.cc:347] Connected to a master server at 127.0.97.62:43437
I20260620 14:47:43.864413 4456 heartbeater.cc:464] Registering TS with master...
I20260620 14:47:43.865134 4456 heartbeater.cc:511] Master 127.0.97.62:43437 requested a full tablet report, sending...
I20260620 14:47:43.866617 4000 ts_manager.cc:194] Registered new tserver with Master: 9a471ae88ebf43aeb2e91ecc41a74ff9 (127.0.97.3:45373)
I20260620 14:47:43.867373 4000 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.97.3@KRBTEST.COM'} at 127.0.97.3:48025
I20260620 14:47:43.870927 388 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260620 14:47:43.878134 4189 heartbeater.cc:503] Master 127.0.97.62:43437 was elected leader, sending a full tablet report...
Jun 20 14:47:43 dist-test-slave-q9j6 krb5kdc[3955](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781966863, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
Jun 20 14:47:43 dist-test-slave-q9j6 krb5kdc[3955](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781966863, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.1@KRBTEST.COM
I20260620 14:47:43.919103 4463 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:47:43.904392 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52094 (local address 127.0.97.1:44803)
0620 14:47:43.905010 (+ 618us) server_negotiation.cc:206] Beginning negotiation
0620 14:47:43.905018 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0620 14:47:43.905045 (+ 27us) server_negotiation.cc:407] Connection header received
0620 14:47:43.905113 (+ 68us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:47:43.905119 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:47:43.905339 (+ 220us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:47:43.905489 (+ 150us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:47:43.906804 (+ 1315us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:47:43.907932 (+ 1128us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:47:43.909110 (+ 1178us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:47:43.909701 (+ 591us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:47:43.909797 (+ 96us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:47:43.913632 (+ 3835us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 14:47:43.913645 (+ 13us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 14:47:43.913662 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 14:47:43.913703 (+ 41us) server_negotiation.cc:922] calling sasl_server_start()
0620 14:47:43.916633 (+ 2930us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:47:43.917274 (+ 641us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:47:43.917281 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:47:43.917283 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:47:43.917399 (+ 116us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:47:43.917707 (+ 308us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:47:43.917710 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:47:43.917712 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:47:43.918046 (+ 334us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 14:47:43.918248 (+ 202us) server_negotiation.cc:1035] Waiting for connection context
0620 14:47:43.918508 (+ 260us) server_negotiation.cc:299] Negotiation successful
0620 14:47:43.918693 (+ 185us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":477,"thread_start_us":319,"threads_started":1}
Jun 20 14:47:43 dist-test-slave-q9j6 krb5kdc[3955](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781966863, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260620 14:47:43.936367 4191 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:47:43.923312 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56916 (local address 127.0.97.62:43437)
0620 14:47:43.923534 (+ 222us) server_negotiation.cc:206] Beginning negotiation
0620 14:47:43.923547 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0620 14:47:43.923711 (+ 164us) server_negotiation.cc:407] Connection header received
0620 14:47:43.923843 (+ 132us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:47:43.923846 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:47:43.923907 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:47:43.923987 (+ 80us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:47:43.925081 (+ 1094us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:47:43.925775 (+ 694us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:47:43.926901 (+ 1126us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:47:43.927284 (+ 383us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:47:43.927370 (+ 86us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:47:43.931161 (+ 3791us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 14:47:43.931166 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 14:47:43.931169 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 14:47:43.931202 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0620 14:47:43.933672 (+ 2470us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:47:43.934352 (+ 680us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:47:43.934357 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:47:43.934361 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:47:43.934483 (+ 122us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:47:43.934878 (+ 395us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:47:43.934883 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:47:43.934886 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:47:43.935207 (+ 321us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 14:47:43.935469 (+ 262us) server_negotiation.cc:1035] Waiting for connection context
0620 14:47:43.935817 (+ 348us) server_negotiation.cc:299] Negotiation successful
0620 14:47:43.936037 (+ 220us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":79}
W20260620 14:47:43.936995 4000 server_base.cc:1116] Unauthorized access attempt to method kudu.master.MasterService.TSHeartbeat from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:56916
I20260620 14:47:43.938112 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskUci3DU/build/asan/bin/kudu pid 4061
I20260620 14:47:43.950135 4463 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:47:43.939032 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52098 (local address 127.0.97.1:44803)
0620 14:47:43.939197 (+ 165us) server_negotiation.cc:206] Beginning negotiation
0620 14:47:43.939202 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 14:47:43.939427 (+ 225us) server_negotiation.cc:407] Connection header received
0620 14:47:43.939552 (+ 125us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:47:43.939556 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:47:43.939710 (+ 154us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:47:43.939798 (+ 88us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:47:43.940913 (+ 1115us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:47:43.941590 (+ 677us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:47:43.942782 (+ 1192us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:47:43.943217 (+ 435us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:47:43.943317 (+ 100us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:47:43.945521 (+ 2204us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 14:47:43.945526 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 14:47:43.945530 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 14:47:43.945561 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0620 14:47:43.948025 (+ 2464us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:47:43.948603 (+ 578us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:47:43.948607 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:47:43.948610 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:47:43.948722 (+ 112us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:47:43.949072 (+ 350us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:47:43.949075 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:47:43.949077 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:47:43.949316 (+ 239us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 14:47:43.949495 (+ 179us) server_negotiation.cc:1035] Waiting for connection context
0620 14:47:43.949748 (+ 253us) server_negotiation.cc:299] Negotiation successful
0620 14:47:43.949913 (+ 165us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":58}
I20260620 14:47:43.950731 4184 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 14:47:44.035634 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskUci3DU/build/asan/bin/kudu with pid 4061
I20260620 14:47:44.051777 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskUci3DU/build/asan/bin/kudu pid 4195
Jun 20 14:47:44 dist-test-slave-q9j6 krb5kdc[3955](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781966863, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.2@KRBTEST.COM
I20260620 14:47:44.067759 4468 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:47:44.052706 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41854 (local address 127.0.97.2:38457)
0620 14:47:44.053250 (+ 544us) server_negotiation.cc:206] Beginning negotiation
0620 14:47:44.053258 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0620 14:47:44.053283 (+ 25us) server_negotiation.cc:407] Connection header received
0620 14:47:44.053347 (+ 64us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:47:44.053354 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:47:44.053575 (+ 221us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:47:44.053715 (+ 140us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:47:44.054978 (+ 1263us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:47:44.056083 (+ 1105us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:47:44.057464 (+ 1381us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:47:44.058016 (+ 552us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:47:44.058115 (+ 99us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:47:44.062196 (+ 4081us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 14:47:44.062209 (+ 13us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 14:47:44.062224 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 14:47:44.062263 (+ 39us) server_negotiation.cc:922] calling sasl_server_start()
0620 14:47:44.065232 (+ 2969us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:47:44.065883 (+ 651us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:47:44.065890 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:47:44.065892 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:47:44.066033 (+ 141us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:47:44.066387 (+ 354us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:47:44.066390 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:47:44.066392 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:47:44.066712 (+ 320us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 14:47:44.066928 (+ 216us) server_negotiation.cc:1035] Waiting for connection context
0620 14:47:44.067176 (+ 248us) server_negotiation.cc:299] Negotiation successful
0620 14:47:44.067390 (+ 214us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":405,"thread_start_us":295,"threads_started":1}
I20260620 14:47:44.069039 4318 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 14:47:44.151202 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskUci3DU/build/asan/bin/kudu with pid 4195
I20260620 14:47:44.167742 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskUci3DU/build/asan/bin/kudu pid 4328
Jun 20 14:47:44 dist-test-slave-q9j6 krb5kdc[3955](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781966863, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.3@KRBTEST.COM
I20260620 14:47:44.184059 4470 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:47:44.168767 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35404 (local address 127.0.97.3:45373)
0620 14:47:44.169391 (+ 624us) server_negotiation.cc:206] Beginning negotiation
0620 14:47:44.169398 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0620 14:47:44.169428 (+ 30us) server_negotiation.cc:407] Connection header received
0620 14:47:44.169488 (+ 60us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:47:44.169495 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:47:44.169714 (+ 219us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:47:44.169843 (+ 129us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:47:44.171042 (+ 1199us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:47:44.172193 (+ 1151us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:47:44.173413 (+ 1220us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:47:44.173965 (+ 552us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:47:44.174067 (+ 102us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:47:44.178266 (+ 4199us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 14:47:44.178282 (+ 16us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 14:47:44.178297 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 14:47:44.178356 (+ 59us) server_negotiation.cc:922] calling sasl_server_start()
0620 14:47:44.181604 (+ 3248us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:47:44.182241 (+ 637us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:47:44.182249 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:47:44.182252 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:47:44.182426 (+ 174us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:47:44.182734 (+ 308us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:47:44.182737 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:47:44.182739 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:47:44.183068 (+ 329us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 14:47:44.183310 (+ 242us) server_negotiation.cc:1035] Waiting for connection context
0620 14:47:44.183497 (+ 187us) server_negotiation.cc:299] Negotiation successful
0620 14:47:44.183781 (+ 284us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":458,"thread_start_us":298,"threads_started":1}
I20260620 14:47:44.185341 4451 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 14:47:44.269517 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskUci3DU/build/asan/bin/kudu with pid 4328
I20260620 14:47:44.286185 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskUci3DU/build/asan/bin/kudu pid 3968
I20260620 14:47:44.287492 4030 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 14:47:44.390072 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskUci3DU/build/asan/bin/kudu with pid 3968
2026-06-20T14:47:44Z chronyd exiting
[ OK ] SecurityITest.TestAuthorizedSuperuser (3872 ms)
[ RUN ] SecurityITest.TestDisableWebUI
Loading random data
Initializing database '/tmp/dist-test-taskUci3DU/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781966829091300-388-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 20 14:47:44 dist-test-slave-q9j6 krb5kdc[4475](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 20 14:47:44 dist-test-slave-q9j6 krb5kdc[4475](info): set up 2 sockets
Jun 20 14:47:44 dist-test-slave-q9j6 krb5kdc[4475](info): commencing operation
krb5kdc: starting...
W20260620 14:47:45.484623 388 mini_kdc.cc:121] Time spent starting KDC: real 1.052s user 0.001s sys 0.017s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskUci3DU/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781966829091300-388-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskUci3DU/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781966829091300-388-0/krb5kdc/test-user.keytab.
Jun 20 14:47:45 dist-test-slave-q9j6 krb5kdc[4475](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781966865, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-20T14:47:45Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T14:47:45Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.97.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskUci3DU/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781966829091300-388-0/krb5kdc/kudu_127.0.97.62.keytab.
Entry for principal kudu/127.0.97.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskUci3DU/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781966829091300-388-0/krb5kdc/kudu_127.0.97.62.keytab.
I20260620 14:47:45.694747 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskUci3DU/build/asan/bin/kudu
/tmp/dist-test-taskUci3DU/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskUci3DU/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781966829091300-388-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskUci3DU/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781966829091300-388-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskUci3DU/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781966829091300-388-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskUci3DU/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781966829091300-388-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:44137
--webserver_interface=127.0.97.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskUci3DU/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781966829091300-388-0/krb5kdc/kudu_127.0.97.62.keytab
--principal=kudu/127.0.97.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:37195
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.97.62:44137
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--webserver_enabled=0 with env {KRB5CCNAME=/tmp/dist-test-taskUci3DU/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781966829091300-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskUci3DU/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781966829091300-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskUci3DU/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781966829091300-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 14:47:46.021029 4488 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 14:47:46.021399 4488 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 14:47:46.021570 4488 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 14:47:46.028940 4488 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 14:47:46.029112 4488 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 14:47:46.029229 4488 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 14:47:46.029332 4488 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 14:47:46.029413 4488 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 14:47:46.039534 4488 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:37195
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskUci3DU/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781966829091300-388-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskUci3DU/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781966829091300-388-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.97.62:44137
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskUci3DU/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781966829091300-388-0/krb5kdc/kudu_127.0.97.62.keytab
--principal=kudu/127.0.97.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:44137
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--webserver_enabled=false
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskUci3DU/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781966829091300-388-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.97.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskUci3DU/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781966829091300-388-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 13:41:20 UTC on ccycloud.quasar-vdagoi.root.comops.site
build id 4457462
ASAN enabled
I20260620 14:47:46.040966 4488 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 14:47:46.042353 4488 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 14:47:46.054472 4496 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 14:47:46.054394 4494 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 14:47:46.054442 4493 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 14:47:46.055341 4488 server_base.cc:1034] running on GCE node
I20260620 14:47:46.056280 4488 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 14:47:46.057768 4488 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 14:47:46.059026 4488 hybrid_clock.cc:648] HybridClock initialized: now 1781966866058987 us; error 56 us; skew 500 ppm
I20260620 14:47:46.059417 4488 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 14:47:46 dist-test-slave-q9j6 krb5kdc[4475](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781966866, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 14:47:46.062670 4488 init.cc:377] Logged in from keytab as kudu/127.0.97.62@KRBTEST.COM (short username kudu)
I20260620 14:47:46.063993 4488 fs_manager.cc:359] Metadata directory not provided
I20260620 14:47:46.064131 4488 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 14:47:46.064492 4488 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 14:47:46.067649 4488 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskUci3DU/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781966829091300-388-0/minicluster-data/master-0/data/instance:
uuid: "f541fbfd0f924d9d81055c7fdfa79538"
format_stamp: "Formatted at 2026-06-20 14:47:46 on dist-test-slave-q9j6"
server_key: "a8397639274c2dd2da214a586b44d7e6"
server_key_iv: "645c4344337ec1990f25264c6e33a1c5"
server_key_version: "encryptionkey@0"
I20260620 14:47:46.068544 4488 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskUci3DU/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781966829091300-388-0/minicluster-data/master-0/wal/instance:
uuid: "f541fbfd0f924d9d81055c7fdfa79538"
format_stamp: "Formatted at 2026-06-20 14:47:46 on dist-test-slave-q9j6"
server_key: "a8397639274c2dd2da214a586b44d7e6"
server_key_iv: "645c4344337ec1990f25264c6e33a1c5"
server_key_version: "encryptionkey@0"
I20260620 14:47:46.073881 4488 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260620 14:47:46.077508 4502 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 14:47:46.078931 4488 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260620 14:47:46.079133 4488 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskUci3DU/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781966829091300-388-0/minicluster-data/master-0/data,/tmp/dist-test-taskUci3DU/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781966829091300-388-0/minicluster-data/master-0/wal
uuid: "f541fbfd0f924d9d81055c7fdfa79538"
format_stamp: "Formatted at 2026-06-20 14:47:46 on dist-test-slave-q9j6"
server_key: "a8397639274c2dd2da214a586b44d7e6"
server_key_iv: "645c4344337ec1990f25264c6e33a1c5"
server_key_version: "encryptionkey@0"
I20260620 14:47:46.079399 4488 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskUci3DU/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781966829091300-388-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskUci3DU/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781966829091300-388-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskUci3DU/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781966829091300-388-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 14:47:46.098112 4488 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 14:47:46.102240 4488 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 14:47:46.102540 4488 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 14:47:46.120132 4488 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.62:44137
I20260620 14:47:46.120150 4553 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.62:44137 every 8 connection(s)
I20260620 14:47:46.121166 4488 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskUci3DU/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781966829091300-388-0/minicluster-data/master-0/data/info.pb
I20260620 14:47:46.124069 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskUci3DU/build/asan/bin/kudu as pid 4488
I20260620 14:47:46.124275 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskUci3DU/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781966829091300-388-0/minicluster-data/master-0/wal/instance
I20260620 14:47:46.124893 388 external_mini_cluster.cc:1428] Setting key 82135c130d6607f8f00b6072416efdcc
I20260620 14:47:46.125142 4554 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 14:47:46.139474 4554 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f541fbfd0f924d9d81055c7fdfa79538: Bootstrap starting.
Jun 20 14:47:46 dist-test-slave-q9j6 krb5kdc[4475](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781966865, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260620 14:47:46.143515 4554 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P f541fbfd0f924d9d81055c7fdfa79538: Neither blocks nor log segments found. Creating new log.
I20260620 14:47:46.144961 4554 log.cc:824] T 00000000000000000000000000000000 P f541fbfd0f924d9d81055c7fdfa79538: Log is configured to *not* fsync() on all Append() calls
I20260620 14:47:46.148526 4554 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f541fbfd0f924d9d81055c7fdfa79538: No bootstrap required, opened a new log
I20260620 14:47:46.150408 4557 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:47:46.127015 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34978 (local address 127.0.97.62:44137)
0620 14:47:46.127900 (+ 885us) server_negotiation.cc:206] Beginning negotiation
0620 14:47:46.127916 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0620 14:47:46.127971 (+ 55us) server_negotiation.cc:407] Connection header received
0620 14:47:46.130106 (+ 2135us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:47:46.130137 (+ 31us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:47:46.130693 (+ 556us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:47:46.131159 (+ 466us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:47:46.132660 (+ 1501us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:47:46.134564 (+ 1904us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:47:46.135985 (+ 1421us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:47:46.137130 (+ 1145us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:47:46.137329 (+ 199us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:47:46.141459 (+ 4130us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 14:47:46.141470 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 14:47:46.141493 (+ 23us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 14:47:46.141541 (+ 48us) server_negotiation.cc:922] calling sasl_server_start()
0620 14:47:46.145396 (+ 3855us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:47:46.146437 (+ 1041us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:47:46.146445 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:47:46.146456 (+ 11us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:47:46.146656 (+ 200us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:47:46.147125 (+ 469us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:47:46.147130 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:47:46.147133 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:47:46.147898 (+ 765us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 14:47:46.148184 (+ 286us) server_negotiation.cc:1035] Waiting for connection context
0620 14:47:46.148600 (+ 416us) server_negotiation.cc:299] Negotiation successful
0620 14:47:46.149057 (+ 457us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":632,"thread_start_us":370,"threads_started":1}
I20260620 14:47:46.157081 4554 raft_consensus.cc:348] T 00000000000000000000000000000000 P f541fbfd0f924d9d81055c7fdfa79538 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f541fbfd0f924d9d81055c7fdfa79538" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 44137 } }
I20260620 14:47:46.157590 4554 raft_consensus.cc:374] T 00000000000000000000000000000000 P f541fbfd0f924d9d81055c7fdfa79538 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 14:47:46.157760 4554 raft_consensus.cc:729] T 00000000000000000000000000000000 P f541fbfd0f924d9d81055c7fdfa79538 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f541fbfd0f924d9d81055c7fdfa79538, State: Initialized, Role: FOLLOWER
I20260620 14:47:46.158347 4554 consensus_queue.cc:260] T 00000000000000000000000000000000 P f541fbfd0f924d9d81055c7fdfa79538 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f541fbfd0f924d9d81055c7fdfa79538" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 44137 } }
I20260620 14:47:46.158589 4554 raft_consensus.cc:388] T 00000000000000000000000000000000 P f541fbfd0f924d9d81055c7fdfa79538 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 14:47:46.158730 4554 raft_consensus.cc:482] T 00000000000000000000000000000000 P f541fbfd0f924d9d81055c7fdfa79538 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 14:47:46.158918 4554 raft_consensus.cc:3037] T 00000000000000000000000000000000 P f541fbfd0f924d9d81055c7fdfa79538 [term 0 FOLLOWER]: Advancing to term 1
I20260620 14:47:46.161180 4554 raft_consensus.cc:504] T 00000000000000000000000000000000 P f541fbfd0f924d9d81055c7fdfa79538 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f541fbfd0f924d9d81055c7fdfa79538" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 44137 } }
I20260620 14:47:46.161695 4554 leader_election.cc:302] T 00000000000000000000000000000000 P f541fbfd0f924d9d81055c7fdfa79538 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: f541fbfd0f924d9d81055c7fdfa79538; no voters:
I20260620 14:47:46.162272 4554 leader_election.cc:288] T 00000000000000000000000000000000 P f541fbfd0f924d9d81055c7fdfa79538 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260620 14:47:46.162434 4559 raft_consensus.cc:2781] T 00000000000000000000000000000000 P f541fbfd0f924d9d81055c7fdfa79538 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 14:47:46.163178 4559 raft_consensus.cc:686] T 00000000000000000000000000000000 P f541fbfd0f924d9d81055c7fdfa79538 [term 1 LEADER]: Becoming Leader. State: Replica: f541fbfd0f924d9d81055c7fdfa79538, State: Running, Role: LEADER
I20260620 14:47:46.163779 4559 consensus_queue.cc:237] T 00000000000000000000000000000000 P f541fbfd0f924d9d81055c7fdfa79538 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f541fbfd0f924d9d81055c7fdfa79538" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 44137 } }
I20260620 14:47:46.164577 4554 sys_catalog.cc:564] T 00000000000000000000000000000000 P f541fbfd0f924d9d81055c7fdfa79538 [sys.catalog]: configured and running, proceeding with master startup.
I20260620 14:47:46.166914 4560 sys_catalog.cc:455] T 00000000000000000000000000000000 P f541fbfd0f924d9d81055c7fdfa79538 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "f541fbfd0f924d9d81055c7fdfa79538" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f541fbfd0f924d9d81055c7fdfa79538" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 44137 } } }
I20260620 14:47:46.166891 4561 sys_catalog.cc:455] T 00000000000000000000000000000000 P f541fbfd0f924d9d81055c7fdfa79538 [sys.catalog]: SysCatalogTable state changed. Reason: New leader f541fbfd0f924d9d81055c7fdfa79538. Latest consensus state: current_term: 1 leader_uuid: "f541fbfd0f924d9d81055c7fdfa79538" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f541fbfd0f924d9d81055c7fdfa79538" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 44137 } } }
I20260620 14:47:46.167421 4561 sys_catalog.cc:458] T 00000000000000000000000000000000 P f541fbfd0f924d9d81055c7fdfa79538 [sys.catalog]: This master's current role is: LEADER
I20260620 14:47:46.167443 4560 sys_catalog.cc:458] T 00000000000000000000000000000000 P f541fbfd0f924d9d81055c7fdfa79538 [sys.catalog]: This master's current role is: LEADER
I20260620 14:47:46.173007 4565 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 14:47:46.178421 4565 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 14:47:46.189181 4565 catalog_manager.cc:1347] Generated new cluster ID: e2f29bec0dce4e9cadf5cdb38a1ef1c1
I20260620 14:47:46.189327 4565 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 14:47:46.205962 4565 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 14:47:46.207298 4565 catalog_manager.cc:1504] Loading token signing keys...
I20260620 14:47:46.214618 4565 catalog_manager.cc:5868] T 00000000000000000000000000000000 P f541fbfd0f924d9d81055c7fdfa79538: Generated new TSK 0
I20260620 14:47:46.215463 4565 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.0.97.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskUci3DU/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781966829091300-388-0/krb5kdc/kudu_127.0.97.1.keytab.
Entry for principal kudu/127.0.97.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskUci3DU/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781966829091300-388-0/krb5kdc/kudu_127.0.97.1.keytab.
I20260620 14:47:46.270792 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskUci3DU/build/asan/bin/kudu
/tmp/dist-test-taskUci3DU/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskUci3DU/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781966829091300-388-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskUci3DU/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781966829091300-388-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskUci3DU/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781966829091300-388-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskUci3DU/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781966829091300-388-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.1:0
--local_ip_for_outbound_sockets=127.0.97.1
--webserver_interface=127.0.97.1
--webserver_port=0
--tserver_master_addrs=127.0.97.62:44137
--keytab_file=/tmp/dist-test-taskUci3DU/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781966829091300-388-0/krb5kdc/kudu_127.0.97.1.keytab
--principal=kudu/127.0.97.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:37195
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--webserver_enabled=0 with env {KRB5CCNAME=/tmp/dist-test-taskUci3DU/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781966829091300-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskUci3DU/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781966829091300-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskUci3DU/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781966829091300-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 14:47:46.625317 4580 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 14:47:46.625608 4580 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 14:47:46.625703 4580 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 14:47:46.633106 4580 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 14:47:46.633222 4580 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 14:47:46.633344 4580 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.1
I20260620 14:47:46.643047 4580 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:37195
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskUci3DU/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781966829091300-388-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskUci3DU/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781966829091300-388-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskUci3DU/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781966829091300-388-0/krb5kdc/kudu_127.0.97.1.keytab
--principal=kudu/127.0.97.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--webserver_enabled=false
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskUci3DU/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781966829091300-388-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.97.1
--webserver_port=0
--tserver_master_addrs=127.0.97.62:44137
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.1
--log_dir=/tmp/dist-test-taskUci3DU/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781966829091300-388-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 13:41:20 UTC on ccycloud.quasar-vdagoi.root.comops.site
build id 4457462
ASAN enabled
I20260620 14:47:46.644361 4580 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 14:47:46.645620 4580 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 14:47:46.658520 4586 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 14:47:46.658567 4588 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 14:47:46.658788 4585 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 14:47:46.659442 4580 server_base.cc:1034] running on GCE node
I20260620 14:47:46.660091 4580 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 14:47:46.661058 4580 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 14:47:46.662257 4580 hybrid_clock.cc:648] HybridClock initialized: now 1781966866662200 us; error 68 us; skew 500 ppm
I20260620 14:47:46.662604 4580 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 14:47:46 dist-test-slave-q9j6 krb5kdc[4475](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781966866, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 14:47:46.666106 4580 init.cc:377] Logged in from keytab as kudu/127.0.97.1@KRBTEST.COM (short username kudu)
I20260620 14:47:46.667332 4580 fs_manager.cc:359] Metadata directory not provided
I20260620 14:47:46.667455 4580 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 14:47:46.667840 4580 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 14:47:46.670836 4580 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskUci3DU/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781966829091300-388-0/minicluster-data/ts-0/data/instance:
uuid: "a17472d5c33f442abf96ac8723ec54fb"
format_stamp: "Formatted at 2026-06-20 14:47:46 on dist-test-slave-q9j6"
server_key: "ed6795b0de20e9e19f5b77637d6fda74"
server_key_iv: "978d6a34be026d1876af8e180be38f6d"
server_key_version: "encryptionkey@0"
I20260620 14:47:46.671805 4580 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskUci3DU/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781966829091300-388-0/minicluster-data/ts-0/wal/instance:
uuid: "a17472d5c33f442abf96ac8723ec54fb"
format_stamp: "Formatted at 2026-06-20 14:47:46 on dist-test-slave-q9j6"
server_key: "ed6795b0de20e9e19f5b77637d6fda74"
server_key_iv: "978d6a34be026d1876af8e180be38f6d"
server_key_version: "encryptionkey@0"
I20260620 14:47:46.677471 4580 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260620 14:47:46.681090 4594 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 14:47:46.682510 4580 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260620 14:47:46.682719 4580 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskUci3DU/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781966829091300-388-0/minicluster-data/ts-0/data,/tmp/dist-test-taskUci3DU/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781966829091300-388-0/minicluster-data/ts-0/wal
uuid: "a17472d5c33f442abf96ac8723ec54fb"
format_stamp: "Formatted at 2026-06-20 14:47:46 on dist-test-slave-q9j6"
server_key: "ed6795b0de20e9e19f5b77637d6fda74"
server_key_iv: "978d6a34be026d1876af8e180be38f6d"
server_key_version: "encryptionkey@0"
I20260620 14:47:46.682955 4580 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskUci3DU/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781966829091300-388-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskUci3DU/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781966829091300-388-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskUci3DU/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781966829091300-388-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 14:47:46.704654 4580 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 14:47:46.709256 4580 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 14:47:46.709558 4580 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 14:47:46.710618 4580 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 14:47:46.712227 4580 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 14:47:46.712358 4580 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 14:47:46.712507 4580 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 14:47:46.712599 4580 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 14:47:46.746338 4580 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.1:34563
I20260620 14:47:46.746378 4706 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.1:34563 every 8 connection(s)
I20260620 14:47:46.747287 4580 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskUci3DU/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781966829091300-388-0/minicluster-data/ts-0/data/info.pb
I20260620 14:47:46.754127 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskUci3DU/build/asan/bin/kudu as pid 4580
I20260620 14:47:46.754325 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskUci3DU/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781966829091300-388-0/minicluster-data/ts-0/wal/instance
I20260620 14:47:46.754967 388 external_mini_cluster.cc:1428] Setting key c74dbf9af40ac3cbb5715d495745f05e
Jun 20 14:47:46 dist-test-slave-q9j6 krb5kdc[4475](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781966866, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.1@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260620 14:47:46.769196 4709 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:47:46.749920 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.1:48905 (local address 127.0.97.62:44137)
0620 14:47:46.750388 (+ 468us) server_negotiation.cc:206] Beginning negotiation
0620 14:47:46.750392 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 14:47:46.751498 (+ 1106us) server_negotiation.cc:407] Connection header received
0620 14:47:46.753473 (+ 1975us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:47:46.753479 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:47:46.753581 (+ 102us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:47:46.753735 (+ 154us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:47:46.755654 (+ 1919us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:47:46.756717 (+ 1063us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:47:46.758485 (+ 1768us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:47:46.758895 (+ 410us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:47:46.759054 (+ 159us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:47:46.762874 (+ 3820us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 14:47:46.762880 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 14:47:46.762883 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 14:47:46.762919 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0620 14:47:46.765837 (+ 2918us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:47:46.766695 (+ 858us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:47:46.766702 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:47:46.766706 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:47:46.766844 (+ 138us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:47:46.767309 (+ 465us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:47:46.767314 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:47:46.767318 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:47:46.767650 (+ 332us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 14:47:46.767814 (+ 164us) server_negotiation.cc:1035] Waiting for connection context
0620 14:47:46.768588 (+ 774us) server_negotiation.cc:299] Negotiation successful
0620 14:47:46.768818 (+ 230us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":363,"thread_start_us":236,"threads_started":1}
I20260620 14:47:46.770349 4710 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:47:46.750568 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:44137 (local address 127.0.97.1:48905)
0620 14:47:46.751301 (+ 733us) negotiation.cc:107] Waiting for socket to connect
0620 14:47:46.751342 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0620 14:47:46.753148 (+ 1806us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 14:47:46.754001 (+ 853us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 14:47:46.754010 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 14:47:46.754709 (+ 699us) client_negotiation.cc:189] Negotiated authn=SASL
0620 14:47:46.755425 (+ 716us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:47:46.755447 (+ 22us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:47:46.756927 (+ 1480us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:47:46.756932 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:47:46.758262 (+ 1330us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:47:46.758277 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:47:46.759165 (+ 888us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:47:46.759173 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:47:46.759441 (+ 268us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:47:46.760248 (+ 807us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 14:47:46.760273 (+ 25us) client_negotiation.cc:656] calling sasl_client_start()
0620 14:47:46.762593 (+ 2320us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 14:47:46.766023 (+ 3430us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 14:47:46.766034 (+ 11us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 14:47:46.766060 (+ 26us) client_negotiation.cc:760] calling sasl_client_step()
0620 14:47:46.766551 (+ 491us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 14:47:46.766968 (+ 417us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 14:47:46.766974 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 14:47:46.766978 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0620 14:47:46.767183 (+ 205us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 14:47:46.767800 (+ 617us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 14:47:46.767810 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 14:47:46.768185 (+ 375us) client_negotiation.cc:769] Sending connection context
0620 14:47:46.768485 (+ 300us) client_negotiation.cc:240] Negotiation successful
0620 14:47:46.769088 (+ 603us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":490,"thread_start_us":305,"threads_started":1}
I20260620 14:47:46.772152 4707 heartbeater.cc:347] Connected to a master server at 127.0.97.62:44137
I20260620 14:47:46.772538 4707 heartbeater.cc:464] Registering TS with master...
WARNING: no policy specified for kudu/127.0.97.2@KRBTEST.COM; defaulting to no policy
I20260620 14:47:46.773447 4707 heartbeater.cc:511] Master 127.0.97.62:44137 requested a full tablet report, sending...
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.2@KRBTEST.COM" created.
I20260620 14:47:46.775974 4519 ts_manager.cc:194] Registered new tserver with Master: a17472d5c33f442abf96ac8723ec54fb (127.0.97.1:34563)
I20260620 14:47:46.777448 4519 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.97.1@KRBTEST.COM'} at 127.0.97.1:48905
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskUci3DU/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781966829091300-388-0/krb5kdc/kudu_127.0.97.2.keytab.
Entry for principal kudu/127.0.97.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskUci3DU/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781966829091300-388-0/krb5kdc/kudu_127.0.97.2.keytab.
I20260620 14:47:46.798993 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskUci3DU/build/asan/bin/kudu
/tmp/dist-test-taskUci3DU/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskUci3DU/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781966829091300-388-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskUci3DU/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781966829091300-388-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskUci3DU/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781966829091300-388-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskUci3DU/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781966829091300-388-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.2:0
--local_ip_for_outbound_sockets=127.0.97.2
--webserver_interface=127.0.97.2
--webserver_port=0
--tserver_master_addrs=127.0.97.62:44137
--keytab_file=/tmp/dist-test-taskUci3DU/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781966829091300-388-0/krb5kdc/kudu_127.0.97.2.keytab
--principal=kudu/127.0.97.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:37195
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--webserver_enabled=0 with env {KRB5CCNAME=/tmp/dist-test-taskUci3DU/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781966829091300-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskUci3DU/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781966829091300-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskUci3DU/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781966829091300-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 14:47:47.130105 4713 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 14:47:47.130405 4713 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 14:47:47.130576 4713 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 14:47:47.138217 4713 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 14:47:47.138343 4713 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 14:47:47.138567 4713 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.2
I20260620 14:47:47.148872 4713 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:37195
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskUci3DU/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781966829091300-388-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskUci3DU/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781966829091300-388-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskUci3DU/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781966829091300-388-0/krb5kdc/kudu_127.0.97.2.keytab
--principal=kudu/127.0.97.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--webserver_enabled=false
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskUci3DU/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781966829091300-388-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.97.2
--webserver_port=0
--tserver_master_addrs=127.0.97.62:44137
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.2
--log_dir=/tmp/dist-test-taskUci3DU/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781966829091300-388-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 13:41:20 UTC on ccycloud.quasar-vdagoi.root.comops.site
build id 4457462
ASAN enabled
I20260620 14:47:47.150307 4713 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 14:47:47.151765 4713 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 14:47:47.165288 4719 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 14:47:47.165288 4721 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 14:47:47.165318 4718 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 14:47:47.166195 4713 server_base.cc:1034] running on GCE node
I20260620 14:47:47.166917 4713 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 14:47:47.167999 4713 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 14:47:47.169236 4713 hybrid_clock.cc:648] HybridClock initialized: now 1781966867169178 us; error 56 us; skew 500 ppm
I20260620 14:47:47.169581 4713 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 14:47:47 dist-test-slave-q9j6 krb5kdc[4475](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781966867, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 14:47:47.172999 4713 init.cc:377] Logged in from keytab as kudu/127.0.97.2@KRBTEST.COM (short username kudu)
I20260620 14:47:47.174309 4713 fs_manager.cc:359] Metadata directory not provided
I20260620 14:47:47.174449 4713 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 14:47:47.174782 4713 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 14:47:47.177917 4713 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskUci3DU/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781966829091300-388-0/minicluster-data/ts-1/data/instance:
uuid: "ed968c45d74e4005be338858498f66d5"
format_stamp: "Formatted at 2026-06-20 14:47:47 on dist-test-slave-q9j6"
server_key: "67100e138309f6f7f37b0da39cd7b2e4"
server_key_iv: "2d80a898dea1570bd5ea4366a2ad52c2"
server_key_version: "encryptionkey@0"
I20260620 14:47:47.178776 4713 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskUci3DU/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781966829091300-388-0/minicluster-data/ts-1/wal/instance:
uuid: "ed968c45d74e4005be338858498f66d5"
format_stamp: "Formatted at 2026-06-20 14:47:47 on dist-test-slave-q9j6"
server_key: "67100e138309f6f7f37b0da39cd7b2e4"
server_key_iv: "2d80a898dea1570bd5ea4366a2ad52c2"
server_key_version: "encryptionkey@0"
I20260620 14:47:47.184614 4713 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.003s sys 0.005s
I20260620 14:47:47.188534 4727 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 14:47:47.189970 4713 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260620 14:47:47.190171 4713 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskUci3DU/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781966829091300-388-0/minicluster-data/ts-1/data,/tmp/dist-test-taskUci3DU/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781966829091300-388-0/minicluster-data/ts-1/wal
uuid: "ed968c45d74e4005be338858498f66d5"
format_stamp: "Formatted at 2026-06-20 14:47:47 on dist-test-slave-q9j6"
server_key: "67100e138309f6f7f37b0da39cd7b2e4"
server_key_iv: "2d80a898dea1570bd5ea4366a2ad52c2"
server_key_version: "encryptionkey@0"
I20260620 14:47:47.190404 4713 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskUci3DU/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781966829091300-388-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskUci3DU/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781966829091300-388-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskUci3DU/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781966829091300-388-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 14:47:47.209450 4713 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 14:47:47.214095 4713 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 14:47:47.214391 4713 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 14:47:47.215382 4713 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 14:47:47.216818 4713 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 14:47:47.216961 4713 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 14:47:47.217087 4713 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 14:47:47.217175 4713 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 14:47:47.250404 4713 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.2:41935
I20260620 14:47:47.250504 4839 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.2:41935 every 8 connection(s)
I20260620 14:47:47.251319 4713 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskUci3DU/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781966829091300-388-0/minicluster-data/ts-1/data/info.pb
I20260620 14:47:47.258780 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskUci3DU/build/asan/bin/kudu as pid 4713
I20260620 14:47:47.258992 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskUci3DU/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781966829091300-388-0/minicluster-data/ts-1/wal/instance
I20260620 14:47:47.259446 388 external_mini_cluster.cc:1428] Setting key 4d3a2439a923dcddd9512789b6fd98ce
Jun 20 14:47:47 dist-test-slave-q9j6 krb5kdc[4475](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781966867, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.2@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260620 14:47:47.273147 4709 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:47:47.253914 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.2:54153 (local address 127.0.97.62:44137)
0620 14:47:47.254099 (+ 185us) server_negotiation.cc:206] Beginning negotiation
0620 14:47:47.254103 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 14:47:47.255339 (+ 1236us) server_negotiation.cc:407] Connection header received
0620 14:47:47.257258 (+ 1919us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:47:47.257264 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:47:47.257362 (+ 98us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:47:47.257475 (+ 113us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:47:47.259098 (+ 1623us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:47:47.259919 (+ 821us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:47:47.262023 (+ 2104us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:47:47.262460 (+ 437us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:47:47.262602 (+ 142us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:47:47.266408 (+ 3806us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 14:47:47.266413 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 14:47:47.266417 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 14:47:47.266455 (+ 38us) server_negotiation.cc:922] calling sasl_server_start()
0620 14:47:47.269188 (+ 2733us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:47:47.270225 (+ 1037us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:47:47.270232 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:47:47.270238 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:47:47.270389 (+ 151us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:47:47.271301 (+ 912us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:47:47.271314 (+ 13us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:47:47.271349 (+ 35us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:47:47.271708 (+ 359us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 14:47:47.271823 (+ 115us) server_negotiation.cc:1035] Waiting for connection context
0620 14:47:47.272735 (+ 912us) server_negotiation.cc:299] Negotiation successful
0620 14:47:47.272902 (+ 167us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":68}
I20260620 14:47:47.274931 4842 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:47:47.254435 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:44137 (local address 127.0.97.2:54153)
0620 14:47:47.255158 (+ 723us) negotiation.cc:107] Waiting for socket to connect
0620 14:47:47.255202 (+ 44us) client_negotiation.cc:174] Beginning negotiation
0620 14:47:47.256960 (+ 1758us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 14:47:47.257723 (+ 763us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 14:47:47.257732 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 14:47:47.258290 (+ 558us) client_negotiation.cc:189] Negotiated authn=SASL
0620 14:47:47.258867 (+ 577us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:47:47.258885 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:47:47.260330 (+ 1445us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:47:47.260335 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:47:47.261827 (+ 1492us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:47:47.261840 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:47:47.262708 (+ 868us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:47:47.262716 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:47:47.262952 (+ 236us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:47:47.263734 (+ 782us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 14:47:47.263762 (+ 28us) client_negotiation.cc:656] calling sasl_client_start()
0620 14:47:47.266163 (+ 2401us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 14:47:47.269352 (+ 3189us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 14:47:47.269365 (+ 13us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 14:47:47.269390 (+ 25us) client_negotiation.cc:760] calling sasl_client_step()
0620 14:47:47.270076 (+ 686us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 14:47:47.270543 (+ 467us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 14:47:47.270551 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 14:47:47.270557 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0620 14:47:47.270793 (+ 236us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 14:47:47.271841 (+ 1048us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 14:47:47.271853 (+ 12us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 14:47:47.272339 (+ 486us) client_negotiation.cc:769] Sending connection context
0620 14:47:47.272672 (+ 333us) client_negotiation.cc:240] Negotiation successful
0620 14:47:47.273303 (+ 631us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":474,"thread_start_us":288,"threads_started":1}
I20260620 14:47:47.276525 4840 heartbeater.cc:347] Connected to a master server at 127.0.97.62:44137
I20260620 14:47:47.276886 4840 heartbeater.cc:464] Registering TS with master...
I20260620 14:47:47.277773 4840 heartbeater.cc:511] Master 127.0.97.62:44137 requested a full tablet report, sending...
I20260620 14:47:47.279481 4519 ts_manager.cc:194] Registered new tserver with Master: ed968c45d74e4005be338858498f66d5 (127.0.97.2:41935)
WARNING: no policy specified for kudu/127.0.97.3@KRBTEST.COM; defaulting to no policy
I20260620 14:47:47.280365 4519 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.97.2@KRBTEST.COM'} at 127.0.97.2:54153
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskUci3DU/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781966829091300-388-0/krb5kdc/kudu_127.0.97.3.keytab.
Entry for principal kudu/127.0.97.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskUci3DU/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781966829091300-388-0/krb5kdc/kudu_127.0.97.3.keytab.
I20260620 14:47:47.306865 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskUci3DU/build/asan/bin/kudu
/tmp/dist-test-taskUci3DU/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskUci3DU/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781966829091300-388-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskUci3DU/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781966829091300-388-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskUci3DU/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781966829091300-388-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskUci3DU/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781966829091300-388-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.3:0
--local_ip_for_outbound_sockets=127.0.97.3
--webserver_interface=127.0.97.3
--webserver_port=0
--tserver_master_addrs=127.0.97.62:44137
--keytab_file=/tmp/dist-test-taskUci3DU/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781966829091300-388-0/krb5kdc/kudu_127.0.97.3.keytab
--principal=kudu/127.0.97.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:37195
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--webserver_enabled=0 with env {KRB5CCNAME=/tmp/dist-test-taskUci3DU/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781966829091300-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskUci3DU/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781966829091300-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskUci3DU/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781966829091300-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 14:47:47.638037 4845 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 14:47:47.638417 4845 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 14:47:47.638600 4845 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 14:47:47.646117 4845 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 14:47:47.646296 4845 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 14:47:47.646494 4845 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.3
I20260620 14:47:47.656970 4845 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:37195
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskUci3DU/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781966829091300-388-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskUci3DU/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781966829091300-388-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskUci3DU/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781966829091300-388-0/krb5kdc/kudu_127.0.97.3.keytab
--principal=kudu/127.0.97.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--webserver_enabled=false
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskUci3DU/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781966829091300-388-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.97.3
--webserver_port=0
--tserver_master_addrs=127.0.97.62:44137
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.3
--log_dir=/tmp/dist-test-taskUci3DU/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781966829091300-388-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 13:41:20 UTC on ccycloud.quasar-vdagoi.root.comops.site
build id 4457462
ASAN enabled
I20260620 14:47:47.658417 4845 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 14:47:47.660010 4845 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 14:47:47.673832 4850 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 14:47:47.673905 4853 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 14:47:47.673852 4851 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 14:47:47.675053 4845 server_base.cc:1034] running on GCE node
I20260620 14:47:47.675849 4845 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 14:47:47.676855 4845 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 14:47:47.678052 4845 hybrid_clock.cc:648] HybridClock initialized: now 1781966867678006 us; error 47 us; skew 500 ppm
I20260620 14:47:47.678429 4845 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 14:47:47 dist-test-slave-q9j6 krb5kdc[4475](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781966867, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 14:47:47.681753 4845 init.cc:377] Logged in from keytab as kudu/127.0.97.3@KRBTEST.COM (short username kudu)
I20260620 14:47:47.683166 4845 fs_manager.cc:359] Metadata directory not provided
I20260620 14:47:47.683305 4845 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 14:47:47.683759 4845 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 14:47:47.686955 4845 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskUci3DU/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781966829091300-388-0/minicluster-data/ts-2/data/instance:
uuid: "807502ed85b349f696cb22cc004f1cf5"
format_stamp: "Formatted at 2026-06-20 14:47:47 on dist-test-slave-q9j6"
server_key: "06d84873817b2c2ce3536b2130af1c1f"
server_key_iv: "43d8ab543b0a693672c093ad506a4092"
server_key_version: "encryptionkey@0"
I20260620 14:47:47.688063 4845 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskUci3DU/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781966829091300-388-0/minicluster-data/ts-2/wal/instance:
uuid: "807502ed85b349f696cb22cc004f1cf5"
format_stamp: "Formatted at 2026-06-20 14:47:47 on dist-test-slave-q9j6"
server_key: "06d84873817b2c2ce3536b2130af1c1f"
server_key_iv: "43d8ab543b0a693672c093ad506a4092"
server_key_version: "encryptionkey@0"
I20260620 14:47:47.693521 4845 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.004s sys 0.000s
I20260620 14:47:47.697383 4859 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 14:47:47.698966 4845 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260620 14:47:47.699164 4845 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskUci3DU/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781966829091300-388-0/minicluster-data/ts-2/data,/tmp/dist-test-taskUci3DU/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781966829091300-388-0/minicluster-data/ts-2/wal
uuid: "807502ed85b349f696cb22cc004f1cf5"
format_stamp: "Formatted at 2026-06-20 14:47:47 on dist-test-slave-q9j6"
server_key: "06d84873817b2c2ce3536b2130af1c1f"
server_key_iv: "43d8ab543b0a693672c093ad506a4092"
server_key_version: "encryptionkey@0"
I20260620 14:47:47.699414 4845 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskUci3DU/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781966829091300-388-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskUci3DU/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781966829091300-388-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskUci3DU/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781966829091300-388-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 14:47:47.724846 4845 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 14:47:47.729239 4845 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 14:47:47.729543 4845 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 14:47:47.730564 4845 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 14:47:47.732043 4845 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 14:47:47.732167 4845 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 14:47:47.732304 4845 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 14:47:47.732398 4845 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 14:47:47.766090 4845 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.3:46535
I20260620 14:47:47.766183 4971 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.3:46535 every 8 connection(s)
I20260620 14:47:47.767019 4845 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskUci3DU/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781966829091300-388-0/minicluster-data/ts-2/data/info.pb
I20260620 14:47:47.777486 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskUci3DU/build/asan/bin/kudu as pid 4845
I20260620 14:47:47.777671 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskUci3DU/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781966829091300-388-0/minicluster-data/ts-2/wal/instance
I20260620 14:47:47.778194 388 external_mini_cluster.cc:1428] Setting key 2cf26259ab510606c979410b1a853635
Jun 20 14:47:47 dist-test-slave-q9j6 krb5kdc[4475](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781966867, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.3@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260620 14:47:47.780746 4707 heartbeater.cc:503] Master 127.0.97.62:44137 was elected leader, sending a full tablet report...
I20260620 14:47:47.787806 4709 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:47:47.769629 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.3:48623 (local address 127.0.97.62:44137)
0620 14:47:47.769833 (+ 204us) server_negotiation.cc:206] Beginning negotiation
0620 14:47:47.769840 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0620 14:47:47.771072 (+ 1232us) server_negotiation.cc:407] Connection header received
0620 14:47:47.772901 (+ 1829us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:47:47.772905 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:47:47.772974 (+ 69us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:47:47.773070 (+ 96us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:47:47.774688 (+ 1618us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:47:47.775457 (+ 769us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:47:47.777100 (+ 1643us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:47:47.777753 (+ 653us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:47:47.777888 (+ 135us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:47:47.781927 (+ 4039us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 14:47:47.781931 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 14:47:47.781935 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 14:47:47.781993 (+ 58us) server_negotiation.cc:922] calling sasl_server_start()
0620 14:47:47.784887 (+ 2894us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:47:47.785602 (+ 715us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:47:47.785606 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:47:47.785608 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:47:47.785733 (+ 125us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:47:47.786160 (+ 427us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:47:47.786165 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:47:47.786168 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:47:47.786456 (+ 288us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 14:47:47.786569 (+ 113us) server_negotiation.cc:1035] Waiting for connection context
0620 14:47:47.787298 (+ 729us) server_negotiation.cc:299] Negotiation successful
0620 14:47:47.787496 (+ 198us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":73}
I20260620 14:47:47.788867 4974 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:47:47.770150 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:44137 (local address 127.0.97.3:48623)
0620 14:47:47.770887 (+ 737us) negotiation.cc:107] Waiting for socket to connect
0620 14:47:47.770943 (+ 56us) client_negotiation.cc:174] Beginning negotiation
0620 14:47:47.772641 (+ 1698us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 14:47:47.773292 (+ 651us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 14:47:47.773300 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 14:47:47.773827 (+ 527us) client_negotiation.cc:189] Negotiated authn=SASL
0620 14:47:47.774507 (+ 680us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:47:47.774527 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:47:47.775663 (+ 1136us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:47:47.775671 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:47:47.776942 (+ 1271us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:47:47.776954 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:47:47.777956 (+ 1002us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:47:47.777961 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:47:47.778154 (+ 193us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:47:47.778936 (+ 782us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 14:47:47.778970 (+ 34us) client_negotiation.cc:656] calling sasl_client_start()
0620 14:47:47.781693 (+ 2723us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 14:47:47.785037 (+ 3344us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 14:47:47.785044 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 14:47:47.785059 (+ 15us) client_negotiation.cc:760] calling sasl_client_step()
0620 14:47:47.785488 (+ 429us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 14:47:47.785823 (+ 335us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 14:47:47.785826 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 14:47:47.785828 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 14:47:47.786013 (+ 185us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 14:47:47.786567 (+ 554us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 14:47:47.786573 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 14:47:47.786911 (+ 338us) client_negotiation.cc:769] Sending connection context
0620 14:47:47.787157 (+ 246us) client_negotiation.cc:240] Negotiation successful
0620 14:47:47.787729 (+ 572us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":492,"thread_start_us":312,"threads_started":1}
I20260620 14:47:47.790340 4972 heartbeater.cc:347] Connected to a master server at 127.0.97.62:44137
I20260620 14:47:47.790654 4972 heartbeater.cc:464] Registering TS with master...
I20260620 14:47:47.791285 4972 heartbeater.cc:511] Master 127.0.97.62:44137 requested a full tablet report, sending...
I20260620 14:47:47.792796 4519 ts_manager.cc:194] Registered new tserver with Master: 807502ed85b349f696cb22cc004f1cf5 (127.0.97.3:46535)
I20260620 14:47:47.793782 4519 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.97.3@KRBTEST.COM'} at 127.0.97.3:48623
I20260620 14:47:47.793812 388 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260620 14:47:47.811161 4709 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:47:47.798409 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34984 (local address 127.0.97.62:44137)
0620 14:47:47.798592 (+ 183us) server_negotiation.cc:206] Beginning negotiation
0620 14:47:47.798600 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0620 14:47:47.799041 (+ 441us) server_negotiation.cc:407] Connection header received
0620 14:47:47.799241 (+ 200us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:47:47.799248 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:47:47.799314 (+ 66us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:47:47.799427 (+ 113us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:47:47.800871 (+ 1444us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:47:47.801573 (+ 702us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:47:47.802895 (+ 1322us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:47:47.803339 (+ 444us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:47:47.803485 (+ 146us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:47:47.805837 (+ 2352us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 14:47:47.805842 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 14:47:47.805846 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 14:47:47.805879 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0620 14:47:47.808505 (+ 2626us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:47:47.809188 (+ 683us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:47:47.809200 (+ 12us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:47:47.809203 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:47:47.809328 (+ 125us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:47:47.809702 (+ 374us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:47:47.809707 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:47:47.809710 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:47:47.810031 (+ 321us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 14:47:47.810292 (+ 261us) server_negotiation.cc:1035] Waiting for connection context
0620 14:47:47.810636 (+ 344us) server_negotiation.cc:299] Negotiation successful
0620 14:47:47.810846 (+ 210us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":53}
I20260620 14:47:47.815464 4519 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:34984:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260620 14:47:47.822376 4519 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260620 14:47:47.858471 4987 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:47:47.835162 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52300 (local address 127.0.97.2:41935)
0620 14:47:47.846672 (+ 11510us) server_negotiation.cc:206] Beginning negotiation
0620 14:47:47.846686 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0620 14:47:47.848502 (+ 1816us) server_negotiation.cc:407] Connection header received
0620 14:47:47.848851 (+ 349us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:47:47.848862 (+ 11us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:47:47.849150 (+ 288us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:47:47.849318 (+ 168us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 14:47:47.850489 (+ 1171us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:47:47.851972 (+ 1483us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:47:47.855318 (+ 3346us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:47:47.856829 (+ 1511us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:47:47.856956 (+ 127us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:47:47.857155 (+ 199us) server_negotiation.cc:1035] Waiting for connection context
0620 14:47:47.857950 (+ 795us) server_negotiation.cc:299] Negotiation successful
0620 14:47:47.858131 (+ 181us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":11241,"thread_start_us":11538,"threads_started":1}
I20260620 14:47:47.858582 4984 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:47:47.834746 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:34563 (local address 127.0.0.1:35442)
0620 14:47:47.848310 (+ 13564us) negotiation.cc:107] Waiting for socket to connect
0620 14:47:47.848370 (+ 60us) client_negotiation.cc:174] Beginning negotiation
0620 14:47:47.848573 (+ 203us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 14:47:47.849612 (+ 1039us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 14:47:47.849618 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 14:47:47.849640 (+ 22us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 14:47:47.849984 (+ 344us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:47:47.849993 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:47:47.851976 (+ 1983us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:47:47.851982 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:47:47.855145 (+ 3163us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:47:47.855153 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:47:47.857690 (+ 2537us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:47:47.857695 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:47:47.857891 (+ 196us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:47:47.857920 (+ 29us) client_negotiation.cc:769] Sending connection context
0620 14:47:47.857993 (+ 73us) client_negotiation.cc:240] Negotiation successful
0620 14:47:47.858181 (+ 188us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":13286,"mutex_wait_us":107,"spinlock_wait_cycles":5504,"thread_start_us":13020,"threads_started":1}
I20260620 14:47:47.859536 4986 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:47:47.834921 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35442 (local address 127.0.97.1:34563)
0620 14:47:47.847504 (+ 12583us) server_negotiation.cc:206] Beginning negotiation
0620 14:47:47.847516 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0620 14:47:47.848774 (+ 1258us) server_negotiation.cc:407] Connection header received
0620 14:47:47.848890 (+ 116us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:47:47.848897 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:47:47.849156 (+ 259us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:47:47.849339 (+ 183us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 14:47:47.850148 (+ 809us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:47:47.851786 (+ 1638us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:47:47.855318 (+ 3532us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:47:47.857160 (+ 1842us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:47:47.857299 (+ 139us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:47:47.857538 (+ 239us) server_negotiation.cc:1035] Waiting for connection context
0620 14:47:47.859014 (+ 1476us) server_negotiation.cc:299] Negotiation successful
0620 14:47:47.859180 (+ 166us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":12367,"thread_start_us":12187,"threads_started":1}
I20260620 14:47:47.859573 4985 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:47:47.834752 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.2:41935 (local address 127.0.0.1:52300)
0620 14:47:47.848310 (+ 13558us) negotiation.cc:107] Waiting for socket to connect
0620 14:47:47.848366 (+ 56us) client_negotiation.cc:174] Beginning negotiation
0620 14:47:47.848615 (+ 249us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 14:47:47.849661 (+ 1046us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 14:47:47.849669 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 14:47:47.849711 (+ 42us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 14:47:47.850260 (+ 549us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:47:47.850276 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:47:47.852124 (+ 1848us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:47:47.852131 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:47:47.855118 (+ 2987us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:47:47.855135 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:47:47.857359 (+ 2224us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:47:47.857366 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:47:47.857634 (+ 268us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:47:47.857704 (+ 70us) client_negotiation.cc:769] Sending connection context
0620 14:47:47.859081 (+ 1377us) client_negotiation.cc:240] Negotiation successful
0620 14:47:47.859209 (+ 128us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":13357,"mutex_wait_us":150,"thread_start_us":13139,"threads_started":1}
I20260620 14:47:47.863797 4775 tablet_service.cc:1467] Processing CreateTablet for tablet b47eb7dc03004f51a2d46781d1b27a9e (DEFAULT_TABLE table=test-table [id=fbf9905f030f48e4bcbbe99c01e3a8c4]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 14:47:47.863797 4642 tablet_service.cc:1467] Processing CreateTablet for tablet b47eb7dc03004f51a2d46781d1b27a9e (DEFAULT_TABLE table=test-table [id=fbf9905f030f48e4bcbbe99c01e3a8c4]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 14:47:47.864568 4989 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:47:47.852978 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.3:46535 (local address 127.0.0.1:50818)
0620 14:47:47.857884 (+ 4906us) negotiation.cc:107] Waiting for socket to connect
0620 14:47:47.857909 (+ 25us) client_negotiation.cc:174] Beginning negotiation
0620 14:47:47.858040 (+ 131us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 14:47:47.858599 (+ 559us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 14:47:47.858603 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 14:47:47.858628 (+ 25us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 14:47:47.858985 (+ 357us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:47:47.859020 (+ 35us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:47:47.860847 (+ 1827us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:47:47.860855 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:47:47.862462 (+ 1607us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:47:47.862474 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:47:47.863770 (+ 1296us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:47:47.863775 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:47:47.863941 (+ 166us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:47:47.863993 (+ 52us) client_negotiation.cc:769] Sending connection context
0620 14:47:47.864112 (+ 119us) client_negotiation.cc:240] Negotiation successful
0620 14:47:47.864233 (+ 121us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":4757,"thread_start_us":4766,"threads_started":1}
I20260620 14:47:47.864696 4988 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:47:47.849069 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50818 (local address 127.0.97.3:46535)
0620 14:47:47.856331 (+ 7262us) server_negotiation.cc:206] Beginning negotiation
0620 14:47:47.856341 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0620 14:47:47.858034 (+ 1693us) server_negotiation.cc:407] Connection header received
0620 14:47:47.858182 (+ 148us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:47:47.858189 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:47:47.858399 (+ 210us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:47:47.858604 (+ 205us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 14:47:47.859209 (+ 605us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:47:47.860702 (+ 1493us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:47:47.862625 (+ 1923us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:47:47.863538 (+ 913us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:47:47.863733 (+ 195us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:47:47.863962 (+ 229us) server_negotiation.cc:1035] Waiting for connection context
0620 14:47:47.864149 (+ 187us) server_negotiation.cc:299] Negotiation successful
0620 14:47:47.864303 (+ 154us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":7083,"thread_start_us":6917,"threads_started":1}
I20260620 14:47:47.865458 4642 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b47eb7dc03004f51a2d46781d1b27a9e. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 14:47:47.865459 4775 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b47eb7dc03004f51a2d46781d1b27a9e. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 14:47:47.868649 4907 tablet_service.cc:1467] Processing CreateTablet for tablet b47eb7dc03004f51a2d46781d1b27a9e (DEFAULT_TABLE table=test-table [id=fbf9905f030f48e4bcbbe99c01e3a8c4]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 14:47:47.869896 4907 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b47eb7dc03004f51a2d46781d1b27a9e. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 14:47:47.881523 4991 tablet_bootstrap.cc:492] T b47eb7dc03004f51a2d46781d1b27a9e P a17472d5c33f442abf96ac8723ec54fb: Bootstrap starting.
I20260620 14:47:47.882437 4990 tablet_bootstrap.cc:492] T b47eb7dc03004f51a2d46781d1b27a9e P ed968c45d74e4005be338858498f66d5: Bootstrap starting.
I20260620 14:47:47.887176 4990 tablet_bootstrap.cc:654] T b47eb7dc03004f51a2d46781d1b27a9e P ed968c45d74e4005be338858498f66d5: Neither blocks nor log segments found. Creating new log.
I20260620 14:47:47.887231 4992 tablet_bootstrap.cc:492] T b47eb7dc03004f51a2d46781d1b27a9e P 807502ed85b349f696cb22cc004f1cf5: Bootstrap starting.
I20260620 14:47:47.889321 4990 log.cc:824] T b47eb7dc03004f51a2d46781d1b27a9e P ed968c45d74e4005be338858498f66d5: Log is configured to *not* fsync() on all Append() calls
I20260620 14:47:47.890196 4991 tablet_bootstrap.cc:654] T b47eb7dc03004f51a2d46781d1b27a9e P a17472d5c33f442abf96ac8723ec54fb: Neither blocks nor log segments found. Creating new log.
I20260620 14:47:47.890714 4992 tablet_bootstrap.cc:654] T b47eb7dc03004f51a2d46781d1b27a9e P 807502ed85b349f696cb22cc004f1cf5: Neither blocks nor log segments found. Creating new log.
I20260620 14:47:47.891479 4991 log.cc:824] T b47eb7dc03004f51a2d46781d1b27a9e P a17472d5c33f442abf96ac8723ec54fb: Log is configured to *not* fsync() on all Append() calls
I20260620 14:47:47.891893 4992 log.cc:824] T b47eb7dc03004f51a2d46781d1b27a9e P 807502ed85b349f696cb22cc004f1cf5: Log is configured to *not* fsync() on all Append() calls
I20260620 14:47:47.892355 4990 tablet_bootstrap.cc:492] T b47eb7dc03004f51a2d46781d1b27a9e P ed968c45d74e4005be338858498f66d5: No bootstrap required, opened a new log
I20260620 14:47:47.892642 4990 ts_tablet_manager.cc:1397] T b47eb7dc03004f51a2d46781d1b27a9e P ed968c45d74e4005be338858498f66d5: Time spent bootstrapping tablet: real 0.010s user 0.001s sys 0.007s
I20260620 14:47:47.898540 4991 tablet_bootstrap.cc:492] T b47eb7dc03004f51a2d46781d1b27a9e P a17472d5c33f442abf96ac8723ec54fb: No bootstrap required, opened a new log
I20260620 14:47:47.898540 4992 tablet_bootstrap.cc:492] T b47eb7dc03004f51a2d46781d1b27a9e P 807502ed85b349f696cb22cc004f1cf5: No bootstrap required, opened a new log
I20260620 14:47:47.898837 4992 ts_tablet_manager.cc:1397] T b47eb7dc03004f51a2d46781d1b27a9e P 807502ed85b349f696cb22cc004f1cf5: Time spent bootstrapping tablet: real 0.012s user 0.004s sys 0.005s
I20260620 14:47:47.898901 4991 ts_tablet_manager.cc:1397] T b47eb7dc03004f51a2d46781d1b27a9e P a17472d5c33f442abf96ac8723ec54fb: Time spent bootstrapping tablet: real 0.018s user 0.008s sys 0.004s
I20260620 14:47:47.901340 4990 raft_consensus.cc:348] T b47eb7dc03004f51a2d46781d1b27a9e P ed968c45d74e4005be338858498f66d5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a17472d5c33f442abf96ac8723ec54fb" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 34563 } } peers { permanent_uuid: "ed968c45d74e4005be338858498f66d5" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 41935 } } peers { permanent_uuid: "807502ed85b349f696cb22cc004f1cf5" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 46535 } }
I20260620 14:47:47.901746 4990 raft_consensus.cc:374] T b47eb7dc03004f51a2d46781d1b27a9e P ed968c45d74e4005be338858498f66d5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 14:47:47.901871 4990 raft_consensus.cc:729] T b47eb7dc03004f51a2d46781d1b27a9e P ed968c45d74e4005be338858498f66d5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ed968c45d74e4005be338858498f66d5, State: Initialized, Role: FOLLOWER
I20260620 14:47:47.902541 4990 consensus_queue.cc:260] T b47eb7dc03004f51a2d46781d1b27a9e P ed968c45d74e4005be338858498f66d5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a17472d5c33f442abf96ac8723ec54fb" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 34563 } } peers { permanent_uuid: "ed968c45d74e4005be338858498f66d5" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 41935 } } peers { permanent_uuid: "807502ed85b349f696cb22cc004f1cf5" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 46535 } }
I20260620 14:47:47.903748 4840 heartbeater.cc:503] Master 127.0.97.62:44137 was elected leader, sending a full tablet report...
I20260620 14:47:47.904466 4990 ts_tablet_manager.cc:1428] T b47eb7dc03004f51a2d46781d1b27a9e P ed968c45d74e4005be338858498f66d5: Time spent starting tablet: real 0.012s user 0.012s sys 0.000s
I20260620 14:47:47.908320 4992 raft_consensus.cc:348] T b47eb7dc03004f51a2d46781d1b27a9e P 807502ed85b349f696cb22cc004f1cf5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a17472d5c33f442abf96ac8723ec54fb" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 34563 } } peers { permanent_uuid: "ed968c45d74e4005be338858498f66d5" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 41935 } } peers { permanent_uuid: "807502ed85b349f696cb22cc004f1cf5" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 46535 } }
I20260620 14:47:47.908782 4992 raft_consensus.cc:374] T b47eb7dc03004f51a2d46781d1b27a9e P 807502ed85b349f696cb22cc004f1cf5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 14:47:47.908900 4992 raft_consensus.cc:729] T b47eb7dc03004f51a2d46781d1b27a9e P 807502ed85b349f696cb22cc004f1cf5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 807502ed85b349f696cb22cc004f1cf5, State: Initialized, Role: FOLLOWER
I20260620 14:47:47.909571 4992 consensus_queue.cc:260] T b47eb7dc03004f51a2d46781d1b27a9e P 807502ed85b349f696cb22cc004f1cf5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a17472d5c33f442abf96ac8723ec54fb" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 34563 } } peers { permanent_uuid: "ed968c45d74e4005be338858498f66d5" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 41935 } } peers { permanent_uuid: "807502ed85b349f696cb22cc004f1cf5" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 46535 } }
I20260620 14:47:47.910708 4972 heartbeater.cc:503] Master 127.0.97.62:44137 was elected leader, sending a full tablet report...
I20260620 14:47:47.911350 4992 ts_tablet_manager.cc:1428] T b47eb7dc03004f51a2d46781d1b27a9e P 807502ed85b349f696cb22cc004f1cf5: Time spent starting tablet: real 0.012s user 0.014s sys 0.000s
I20260620 14:47:47.910238 4991 raft_consensus.cc:348] T b47eb7dc03004f51a2d46781d1b27a9e P a17472d5c33f442abf96ac8723ec54fb [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a17472d5c33f442abf96ac8723ec54fb" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 34563 } } peers { permanent_uuid: "ed968c45d74e4005be338858498f66d5" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 41935 } } peers { permanent_uuid: "807502ed85b349f696cb22cc004f1cf5" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 46535 } }
I20260620 14:47:47.911907 4991 raft_consensus.cc:374] T b47eb7dc03004f51a2d46781d1b27a9e P a17472d5c33f442abf96ac8723ec54fb [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 14:47:47.912055 4991 raft_consensus.cc:729] T b47eb7dc03004f51a2d46781d1b27a9e P a17472d5c33f442abf96ac8723ec54fb [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a17472d5c33f442abf96ac8723ec54fb, State: Initialized, Role: FOLLOWER
I20260620 14:47:47.912787 4991 consensus_queue.cc:260] T b47eb7dc03004f51a2d46781d1b27a9e P a17472d5c33f442abf96ac8723ec54fb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a17472d5c33f442abf96ac8723ec54fb" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 34563 } } peers { permanent_uuid: "ed968c45d74e4005be338858498f66d5" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 41935 } } peers { permanent_uuid: "807502ed85b349f696cb22cc004f1cf5" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 46535 } }
I20260620 14:47:47.914637 4991 ts_tablet_manager.cc:1428] T b47eb7dc03004f51a2d46781d1b27a9e P a17472d5c33f442abf96ac8723ec54fb: Time spent starting tablet: real 0.016s user 0.012s sys 0.001s
W20260620 14:47:47.999933 4708 tablet.cc:2369] T b47eb7dc03004f51a2d46781d1b27a9e P a17472d5c33f442abf96ac8723ec54fb: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260620 14:47:48.003194 4841 tablet.cc:2369] T b47eb7dc03004f51a2d46781d1b27a9e P ed968c45d74e4005be338858498f66d5: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260620 14:47:48.018607 4973 tablet.cc:2369] T b47eb7dc03004f51a2d46781d1b27a9e P 807502ed85b349f696cb22cc004f1cf5: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260620 14:47:48.146071 4998 raft_consensus.cc:482] T b47eb7dc03004f51a2d46781d1b27a9e P a17472d5c33f442abf96ac8723ec54fb [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260620 14:47:48.146360 4998 raft_consensus.cc:504] T b47eb7dc03004f51a2d46781d1b27a9e P a17472d5c33f442abf96ac8723ec54fb [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a17472d5c33f442abf96ac8723ec54fb" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 34563 } } peers { permanent_uuid: "ed968c45d74e4005be338858498f66d5" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 41935 } } peers { permanent_uuid: "807502ed85b349f696cb22cc004f1cf5" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 46535 } }
I20260620 14:47:48.148067 4998 leader_election.cc:288] T b47eb7dc03004f51a2d46781d1b27a9e P a17472d5c33f442abf96ac8723ec54fb [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers ed968c45d74e4005be338858498f66d5 (127.0.97.2:41935), 807502ed85b349f696cb22cc004f1cf5 (127.0.97.3:46535)
I20260620 14:47:48.157536 4987 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:47:48.148484 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.1:36945 (local address 127.0.97.2:41935)
0620 14:47:48.148668 (+ 184us) server_negotiation.cc:206] Beginning negotiation
0620 14:47:48.148675 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0620 14:47:48.149622 (+ 947us) server_negotiation.cc:407] Connection header received
0620 14:47:48.149739 (+ 117us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:47:48.149745 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:47:48.149822 (+ 77us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:47:48.149924 (+ 102us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 14:47:48.150734 (+ 810us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:47:48.151856 (+ 1122us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:47:48.155100 (+ 3244us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:47:48.156383 (+ 1283us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:47:48.156527 (+ 144us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:47:48.156632 (+ 105us) server_negotiation.cc:1035] Waiting for connection context
0620 14:47:48.157131 (+ 499us) server_negotiation.cc:299] Negotiation successful
0620 14:47:48.157264 (+ 133us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":54}
I20260620 14:47:48.157536 4988 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:47:48.148477 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.1:35613 (local address 127.0.97.3:46535)
0620 14:47:48.148668 (+ 191us) server_negotiation.cc:206] Beginning negotiation
0620 14:47:48.148674 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 14:47:48.149533 (+ 859us) server_negotiation.cc:407] Connection header received
0620 14:47:48.149741 (+ 208us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:47:48.149745 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:47:48.149822 (+ 77us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:47:48.149923 (+ 101us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 14:47:48.150735 (+ 812us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:47:48.151857 (+ 1122us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:47:48.155104 (+ 3247us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:47:48.156383 (+ 1279us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:47:48.156527 (+ 144us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:47:48.156632 (+ 105us) server_negotiation.cc:1035] Waiting for connection context
0620 14:47:48.157131 (+ 499us) server_negotiation.cc:299] Negotiation successful
0620 14:47:48.157264 (+ 133us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":59}
I20260620 14:47:48.157775 4999 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:47:48.148334 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.3:46535 (local address 127.0.97.1:35613)
0620 14:47:48.149405 (+ 1071us) negotiation.cc:107] Waiting for socket to connect
0620 14:47:48.149426 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0620 14:47:48.149594 (+ 168us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 14:47:48.149959 (+ 365us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 14:47:48.149964 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 14:47:48.150005 (+ 41us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 14:47:48.150594 (+ 589us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:47:48.150602 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:47:48.152025 (+ 1423us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:47:48.152032 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:47:48.154904 (+ 2872us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:47:48.154923 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:47:48.156676 (+ 1753us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:47:48.156681 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:47:48.156905 (+ 224us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:47:48.156984 (+ 79us) client_negotiation.cc:769] Sending connection context
0620 14:47:48.157082 (+ 98us) client_negotiation.cc:240] Negotiation successful
0620 14:47:48.157269 (+ 187us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":902,"mutex_wait_us":370,"spinlock_wait_cycles":3328,"thread_start_us":542,"threads_started":1}
I20260620 14:47:48.157796 5000 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:47:48.148426 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.2:41935 (local address 127.0.97.1:36945)
0620 14:47:48.149514 (+ 1088us) negotiation.cc:107] Waiting for socket to connect
0620 14:47:48.149528 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0620 14:47:48.149618 (+ 90us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 14:47:48.149959 (+ 341us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 14:47:48.149964 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 14:47:48.149995 (+ 31us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 14:47:48.150525 (+ 530us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:47:48.150551 (+ 26us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:47:48.152025 (+ 1474us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:47:48.152032 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:47:48.154904 (+ 2872us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:47:48.154923 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:47:48.156653 (+ 1730us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:47:48.156665 (+ 12us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:47:48.156899 (+ 234us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:47:48.156981 (+ 82us) client_negotiation.cc:769] Sending connection context
0620 14:47:48.157082 (+ 101us) client_negotiation.cc:240] Negotiation successful
0620 14:47:48.157265 (+ 183us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":970,"mutex_wait_us":331,"thread_start_us":490,"threads_started":1}
I20260620 14:47:48.158947 4927 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b47eb7dc03004f51a2d46781d1b27a9e" candidate_uuid: "a17472d5c33f442abf96ac8723ec54fb" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "807502ed85b349f696cb22cc004f1cf5" is_pre_election: true
I20260620 14:47:48.158947 4795 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b47eb7dc03004f51a2d46781d1b27a9e" candidate_uuid: "a17472d5c33f442abf96ac8723ec54fb" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ed968c45d74e4005be338858498f66d5" is_pre_election: true
I20260620 14:47:48.159466 4927 raft_consensus.cc:2454] T b47eb7dc03004f51a2d46781d1b27a9e P 807502ed85b349f696cb22cc004f1cf5 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a17472d5c33f442abf96ac8723ec54fb in term 0.
I20260620 14:47:48.159469 4795 raft_consensus.cc:2454] T b47eb7dc03004f51a2d46781d1b27a9e P ed968c45d74e4005be338858498f66d5 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a17472d5c33f442abf96ac8723ec54fb in term 0.
I20260620 14:47:48.160207 4595 leader_election.cc:302] T b47eb7dc03004f51a2d46781d1b27a9e P a17472d5c33f442abf96ac8723ec54fb [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 3 yes votes; 0 no votes. yes voters: 807502ed85b349f696cb22cc004f1cf5, a17472d5c33f442abf96ac8723ec54fb, ed968c45d74e4005be338858498f66d5; no voters:
I20260620 14:47:48.160687 4998 raft_consensus.cc:2781] T b47eb7dc03004f51a2d46781d1b27a9e P a17472d5c33f442abf96ac8723ec54fb [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260620 14:47:48.160878 4998 raft_consensus.cc:482] T b47eb7dc03004f51a2d46781d1b27a9e P a17472d5c33f442abf96ac8723ec54fb [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260620 14:47:48.160985 4998 raft_consensus.cc:3037] T b47eb7dc03004f51a2d46781d1b27a9e P a17472d5c33f442abf96ac8723ec54fb [term 0 FOLLOWER]: Advancing to term 1
I20260620 14:47:48.163136 4998 raft_consensus.cc:504] T b47eb7dc03004f51a2d46781d1b27a9e P a17472d5c33f442abf96ac8723ec54fb [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a17472d5c33f442abf96ac8723ec54fb" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 34563 } } peers { permanent_uuid: "ed968c45d74e4005be338858498f66d5" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 41935 } } peers { permanent_uuid: "807502ed85b349f696cb22cc004f1cf5" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 46535 } }
I20260620 14:47:48.163815 4998 leader_election.cc:288] T b47eb7dc03004f51a2d46781d1b27a9e P a17472d5c33f442abf96ac8723ec54fb [CANDIDATE]: Term 1 election: Requested vote from peers ed968c45d74e4005be338858498f66d5 (127.0.97.2:41935), 807502ed85b349f696cb22cc004f1cf5 (127.0.97.3:46535)
I20260620 14:47:48.164261 4795 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b47eb7dc03004f51a2d46781d1b27a9e" candidate_uuid: "a17472d5c33f442abf96ac8723ec54fb" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ed968c45d74e4005be338858498f66d5"
I20260620 14:47:48.164366 4927 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b47eb7dc03004f51a2d46781d1b27a9e" candidate_uuid: "a17472d5c33f442abf96ac8723ec54fb" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "807502ed85b349f696cb22cc004f1cf5"
I20260620 14:47:48.164491 4795 raft_consensus.cc:3037] T b47eb7dc03004f51a2d46781d1b27a9e P ed968c45d74e4005be338858498f66d5 [term 0 FOLLOWER]: Advancing to term 1
I20260620 14:47:48.164613 4927 raft_consensus.cc:3037] T b47eb7dc03004f51a2d46781d1b27a9e P 807502ed85b349f696cb22cc004f1cf5 [term 0 FOLLOWER]: Advancing to term 1
I20260620 14:47:48.166746 4795 raft_consensus.cc:2454] T b47eb7dc03004f51a2d46781d1b27a9e P ed968c45d74e4005be338858498f66d5 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a17472d5c33f442abf96ac8723ec54fb in term 1.
I20260620 14:47:48.166903 4927 raft_consensus.cc:2454] T b47eb7dc03004f51a2d46781d1b27a9e P 807502ed85b349f696cb22cc004f1cf5 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a17472d5c33f442abf96ac8723ec54fb in term 1.
I20260620 14:47:48.167164 4595 leader_election.cc:302] T b47eb7dc03004f51a2d46781d1b27a9e P a17472d5c33f442abf96ac8723ec54fb [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: a17472d5c33f442abf96ac8723ec54fb, ed968c45d74e4005be338858498f66d5; no voters:
I20260620 14:47:48.167454 4998 raft_consensus.cc:2781] T b47eb7dc03004f51a2d46781d1b27a9e P a17472d5c33f442abf96ac8723ec54fb [term 1 FOLLOWER]: Leader election won for term 1
I20260620 14:47:48.167941 4998 raft_consensus.cc:686] T b47eb7dc03004f51a2d46781d1b27a9e P a17472d5c33f442abf96ac8723ec54fb [term 1 LEADER]: Becoming Leader. State: Replica: a17472d5c33f442abf96ac8723ec54fb, State: Running, Role: LEADER
I20260620 14:47:48.168434 4998 consensus_queue.cc:237] T b47eb7dc03004f51a2d46781d1b27a9e P a17472d5c33f442abf96ac8723ec54fb [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a17472d5c33f442abf96ac8723ec54fb" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 34563 } } peers { permanent_uuid: "ed968c45d74e4005be338858498f66d5" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 41935 } } peers { permanent_uuid: "807502ed85b349f696cb22cc004f1cf5" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 46535 } }
I20260620 14:47:48.173038 4518 catalog_manager.cc:5495] T b47eb7dc03004f51a2d46781d1b27a9e P a17472d5c33f442abf96ac8723ec54fb reported cstate change: term changed from 0 to 1, leader changed from <none> to a17472d5c33f442abf96ac8723ec54fb (127.0.97.1). New cstate: current_term: 1 leader_uuid: "a17472d5c33f442abf96ac8723ec54fb" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a17472d5c33f442abf96ac8723ec54fb" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 34563 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "ed968c45d74e4005be338858498f66d5" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 41935 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "807502ed85b349f696cb22cc004f1cf5" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 46535 } health_report { overall_health: UNKNOWN } } }
I20260620 14:47:48.227509 4986 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:47:48.222031 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35450 (local address 127.0.97.1:34563)
0620 14:47:48.222252 (+ 221us) server_negotiation.cc:206] Beginning negotiation
0620 14:47:48.222259 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0620 14:47:48.222279 (+ 20us) server_negotiation.cc:407] Connection header received
0620 14:47:48.222506 (+ 227us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:47:48.222510 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:47:48.222586 (+ 76us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:47:48.222745 (+ 159us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 14:47:48.223135 (+ 390us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:47:48.223909 (+ 774us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:47:48.225217 (+ 1308us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:47:48.225714 (+ 497us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:47:48.225850 (+ 136us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:47:48.226349 (+ 499us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 14:47:48.226938 (+ 589us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0620 14:47:48.227105 (+ 167us) server_negotiation.cc:1035] Waiting for connection context
0620 14:47:48.227156 (+ 51us) server_negotiation.cc:299] Negotiation successful
0620 14:47:48.227235 (+ 79us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":75}
I20260620 14:47:48.235793 4927 raft_consensus.cc:1261] T b47eb7dc03004f51a2d46781d1b27a9e P 807502ed85b349f696cb22cc004f1cf5 [term 1 FOLLOWER]: Refusing update from remote peer a17472d5c33f442abf96ac8723ec54fb: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 14:47:48.235796 4795 raft_consensus.cc:1261] T b47eb7dc03004f51a2d46781d1b27a9e P ed968c45d74e4005be338858498f66d5 [term 1 FOLLOWER]: Refusing update from remote peer a17472d5c33f442abf96ac8723ec54fb: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 14:47:48.236821 5001 consensus_queue.cc:1035] T b47eb7dc03004f51a2d46781d1b27a9e P a17472d5c33f442abf96ac8723ec54fb [LEADER]: Connected to new peer: Peer: permanent_uuid: "ed968c45d74e4005be338858498f66d5" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 41935 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 14:47:48.237175 4998 consensus_queue.cc:1035] T b47eb7dc03004f51a2d46781d1b27a9e P a17472d5c33f442abf96ac8723ec54fb [LEADER]: Connected to new peer: Peer: permanent_uuid: "807502ed85b349f696cb22cc004f1cf5" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 46535 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 14:47:48.249722 5006 mvcc.cc:205] Tried to move back new op lower bound from 7298936292283731968 to 7298936292021424128. Current Snapshot: MvccSnapshot[applied={T|T < 7298936292283731968}]
I20260620 14:47:48.265314 4518 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:34984:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 14:47:48.265609 4518 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:34984:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 14:47:48.269452 4518 catalog_manager.cc:5782] T 00000000000000000000000000000000 P f541fbfd0f924d9d81055c7fdfa79538: Sending DeleteTablet for 3 replicas of tablet b47eb7dc03004f51a2d46781d1b27a9e
I20260620 14:47:48.270762 4775 tablet_service.cc:1514] Processing DeleteTablet for tablet b47eb7dc03004f51a2d46781d1b27a9e with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 14:47:48 UTC) from {username='kudu', principal='kudu/127.0.97.62@KRBTEST.COM'} at 127.0.0.1:52300
I20260620 14:47:48.270762 4907 tablet_service.cc:1514] Processing DeleteTablet for tablet b47eb7dc03004f51a2d46781d1b27a9e with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 14:47:48 UTC) from {username='kudu', principal='kudu/127.0.97.62@KRBTEST.COM'} at 127.0.0.1:50818
I20260620 14:47:48.270884 4642 tablet_service.cc:1514] Processing DeleteTablet for tablet b47eb7dc03004f51a2d46781d1b27a9e with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 14:47:48 UTC) from {username='kudu', principal='kudu/127.0.97.62@KRBTEST.COM'} at 127.0.0.1:35442
I20260620 14:47:48.275331 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskUci3DU/build/asan/bin/kudu pid 4580
I20260620 14:47:48.283290 5016 tablet_replica.cc:330] T b47eb7dc03004f51a2d46781d1b27a9e P a17472d5c33f442abf96ac8723ec54fb: stopping tablet replica
I20260620 14:47:48.283962 5016 raft_consensus.cc:2229] T b47eb7dc03004f51a2d46781d1b27a9e P a17472d5c33f442abf96ac8723ec54fb [term 1 LEADER]: Raft consensus shutting down.
I20260620 14:47:48.284793 5016 raft_consensus.cc:2258] T b47eb7dc03004f51a2d46781d1b27a9e P a17472d5c33f442abf96ac8723ec54fb [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 14:47:48.287329 5016 ts_tablet_manager.cc:1905] T b47eb7dc03004f51a2d46781d1b27a9e P a17472d5c33f442abf96ac8723ec54fb: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 14:47:48.295421 5016 ts_tablet_manager.cc:1918] T b47eb7dc03004f51a2d46781d1b27a9e P a17472d5c33f442abf96ac8723ec54fb: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260620 14:47:48.295732 5016 log.cc:1196] T b47eb7dc03004f51a2d46781d1b27a9e P a17472d5c33f442abf96ac8723ec54fb: Deleting WAL directory at /tmp/dist-test-taskUci3DU/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781966829091300-388-0/minicluster-data/ts-0/wal/wals/b47eb7dc03004f51a2d46781d1b27a9e
I20260620 14:47:48.296471 5016 ts_tablet_manager.cc:1939] T b47eb7dc03004f51a2d46781d1b27a9e P a17472d5c33f442abf96ac8723ec54fb: Deleting consensus metadata
I20260620 14:47:48.298501 4506 catalog_manager.cc:4841] TS a17472d5c33f442abf96ac8723ec54fb (127.0.97.1:34563): tablet b47eb7dc03004f51a2d46781d1b27a9e (table test-table [id=fbf9905f030f48e4bcbbe99c01e3a8c4]) successfully deleted
I20260620 14:47:48.299960 5015 tablet_replica.cc:330] T b47eb7dc03004f51a2d46781d1b27a9e P ed968c45d74e4005be338858498f66d5: stopping tablet replica
I20260620 14:47:48.300760 5015 raft_consensus.cc:2229] T b47eb7dc03004f51a2d46781d1b27a9e P ed968c45d74e4005be338858498f66d5 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 14:47:48.301541 5015 raft_consensus.cc:2258] T b47eb7dc03004f51a2d46781d1b27a9e P ed968c45d74e4005be338858498f66d5 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 14:47:48.302470 5014 tablet_replica.cc:330] T b47eb7dc03004f51a2d46781d1b27a9e P 807502ed85b349f696cb22cc004f1cf5: stopping tablet replica
I20260620 14:47:48.302951 5014 raft_consensus.cc:2229] T b47eb7dc03004f51a2d46781d1b27a9e P 807502ed85b349f696cb22cc004f1cf5 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 14:47:48.303527 5014 raft_consensus.cc:2258] T b47eb7dc03004f51a2d46781d1b27a9e P 807502ed85b349f696cb22cc004f1cf5 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 14:47:48.305035 5015 ts_tablet_manager.cc:1905] T b47eb7dc03004f51a2d46781d1b27a9e P ed968c45d74e4005be338858498f66d5: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 14:47:48.305531 5014 ts_tablet_manager.cc:1905] T b47eb7dc03004f51a2d46781d1b27a9e P 807502ed85b349f696cb22cc004f1cf5: Deleting tablet data with delete state TABLET_DATA_DELETED
Jun 20 14:47:48 dist-test-slave-q9j6 krb5kdc[4475](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781966865, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.1@KRBTEST.COM
I20260620 14:47:48.312314 5014 ts_tablet_manager.cc:1918] T b47eb7dc03004f51a2d46781d1b27a9e P 807502ed85b349f696cb22cc004f1cf5: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260620 14:47:48.312315 5015 ts_tablet_manager.cc:1918] T b47eb7dc03004f51a2d46781d1b27a9e P ed968c45d74e4005be338858498f66d5: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260620 14:47:48.312503 5015 log.cc:1196] T b47eb7dc03004f51a2d46781d1b27a9e P ed968c45d74e4005be338858498f66d5: Deleting WAL directory at /tmp/dist-test-taskUci3DU/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781966829091300-388-0/minicluster-data/ts-1/wal/wals/b47eb7dc03004f51a2d46781d1b27a9e
I20260620 14:47:48.312664 5014 log.cc:1196] T b47eb7dc03004f51a2d46781d1b27a9e P 807502ed85b349f696cb22cc004f1cf5: Deleting WAL directory at /tmp/dist-test-taskUci3DU/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781966829091300-388-0/minicluster-data/ts-2/wal/wals/b47eb7dc03004f51a2d46781d1b27a9e
I20260620 14:47:48.313098 5015 ts_tablet_manager.cc:1939] T b47eb7dc03004f51a2d46781d1b27a9e P ed968c45d74e4005be338858498f66d5: Deleting consensus metadata
I20260620 14:47:48.313195 5014 ts_tablet_manager.cc:1939] T b47eb7dc03004f51a2d46781d1b27a9e P 807502ed85b349f696cb22cc004f1cf5: Deleting consensus metadata
I20260620 14:47:48.315203 4506 catalog_manager.cc:4841] TS 807502ed85b349f696cb22cc004f1cf5 (127.0.97.3:46535): tablet b47eb7dc03004f51a2d46781d1b27a9e (table test-table [id=fbf9905f030f48e4bcbbe99c01e3a8c4]) successfully deleted
I20260620 14:47:48.315951 4503 catalog_manager.cc:4841] TS ed968c45d74e4005be338858498f66d5 (127.0.97.2:41935): tablet b47eb7dc03004f51a2d46781d1b27a9e (table test-table [id=fbf9905f030f48e4bcbbe99c01e3a8c4]) successfully deleted
I20260620 14:47:48.318794 4986 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:47:48.279146 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35452 (local address 127.0.97.1:34563)
0620 14:47:48.279386 (+ 240us) server_negotiation.cc:206] Beginning negotiation
0620 14:47:48.279395 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0620 14:47:48.298426 (+ 19031us) server_negotiation.cc:407] Connection header received
0620 14:47:48.298995 (+ 569us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:47:48.299002 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:47:48.299100 (+ 98us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:47:48.299220 (+ 120us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:47:48.301058 (+ 1838us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:47:48.302265 (+ 1207us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:47:48.304659 (+ 2394us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:47:48.305375 (+ 716us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:47:48.305483 (+ 108us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:47:48.310881 (+ 5398us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 14:47:48.310900 (+ 19us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 14:47:48.310921 (+ 21us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 14:47:48.310975 (+ 54us) server_negotiation.cc:922] calling sasl_server_start()
0620 14:47:48.315513 (+ 4538us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:47:48.316597 (+ 1084us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:47:48.316608 (+ 11us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:47:48.316611 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:47:48.316782 (+ 171us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:47:48.317223 (+ 441us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:47:48.317226 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:47:48.317228 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:47:48.317589 (+ 361us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 14:47:48.317810 (+ 221us) server_negotiation.cc:1035] Waiting for connection context
0620 14:47:48.318262 (+ 452us) server_negotiation.cc:299] Negotiation successful
0620 14:47:48.318464 (+ 202us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":71}
I20260620 14:47:48.319420 4702 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 14:47:48.420411 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskUci3DU/build/asan/bin/kudu with pid 4580
I20260620 14:47:48.439383 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskUci3DU/build/asan/bin/kudu pid 4713
Jun 20 14:47:48 dist-test-slave-q9j6 krb5kdc[4475](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781966865, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.2@KRBTEST.COM
I20260620 14:47:48.455044 4987 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:47:48.440180 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52310 (local address 127.0.97.2:41935)
0620 14:47:48.440362 (+ 182us) server_negotiation.cc:206] Beginning negotiation
0620 14:47:48.440368 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 14:47:48.440454 (+ 86us) server_negotiation.cc:407] Connection header received
0620 14:47:48.440653 (+ 199us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:47:48.440660 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:47:48.440728 (+ 68us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:47:48.440836 (+ 108us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:47:48.442011 (+ 1175us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:47:48.442728 (+ 717us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:47:48.444059 (+ 1331us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:47:48.444454 (+ 395us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:47:48.444588 (+ 134us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:47:48.448941 (+ 4353us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 14:47:48.448960 (+ 19us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 14:47:48.448978 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 14:47:48.449024 (+ 46us) server_negotiation.cc:922] calling sasl_server_start()
0620 14:47:48.452139 (+ 3115us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:47:48.452879 (+ 740us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:47:48.452887 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:47:48.452890 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:47:48.453047 (+ 157us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:47:48.453466 (+ 419us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:47:48.453470 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:47:48.453472 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:47:48.453836 (+ 364us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 14:47:48.454132 (+ 296us) server_negotiation.cc:1035] Waiting for connection context
0620 14:47:48.454485 (+ 353us) server_negotiation.cc:299] Negotiation successful
0620 14:47:48.454707 (+ 222us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":65}
I20260620 14:47:48.455799 4835 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 14:47:48.545106 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskUci3DU/build/asan/bin/kudu with pid 4713
I20260620 14:47:48.562373 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskUci3DU/build/asan/bin/kudu pid 4845
Jun 20 14:47:48 dist-test-slave-q9j6 krb5kdc[4475](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781966865, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.3@KRBTEST.COM
I20260620 14:47:48.577335 4988 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:47:48.563137 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50830 (local address 127.0.97.3:46535)
0620 14:47:48.563311 (+ 174us) server_negotiation.cc:206] Beginning negotiation
0620 14:47:48.563316 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 14:47:48.563335 (+ 19us) server_negotiation.cc:407] Connection header received
0620 14:47:48.563514 (+ 179us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:47:48.563518 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:47:48.563584 (+ 66us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:47:48.563739 (+ 155us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:47:48.565120 (+ 1381us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:47:48.565863 (+ 743us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:47:48.567098 (+ 1235us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:47:48.567501 (+ 403us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:47:48.567659 (+ 158us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:47:48.571989 (+ 4330us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 14:47:48.572003 (+ 14us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 14:47:48.572020 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 14:47:48.572062 (+ 42us) server_negotiation.cc:922] calling sasl_server_start()
0620 14:47:48.574786 (+ 2724us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:47:48.575537 (+ 751us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:47:48.575544 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:47:48.575547 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:47:48.575735 (+ 188us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:47:48.576061 (+ 326us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:47:48.576065 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:47:48.576066 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:47:48.576401 (+ 335us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 14:47:48.576624 (+ 223us) server_negotiation.cc:1035] Waiting for connection context
0620 14:47:48.576919 (+ 295us) server_negotiation.cc:299] Negotiation successful
0620 14:47:48.577080 (+ 161us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":63}
I20260620 14:47:48.577998 4967 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 14:47:48.679908 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskUci3DU/build/asan/bin/kudu with pid 4845
I20260620 14:47:48.696375 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskUci3DU/build/asan/bin/kudu pid 4488
I20260620 14:47:48.697582 4549 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 14:47:48.804658 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskUci3DU/build/asan/bin/kudu with pid 4488
2026-06-20T14:47:48Z chronyd exiting
[ OK ] SecurityITest.TestDisableWebUI (4415 ms)
[ RUN ] SecurityITest.TestDisableAuthenticationEncryption
2026-06-20T14:47:48Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T14:47:48Z Disabled control of system clock
I20260620 14:47:48.873272 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskUci3DU/build/asan/bin/kudu
/tmp/dist-test-taskUci3DU/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskUci3DU/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781966829091300-388-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskUci3DU/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781966829091300-388-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskUci3DU/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781966829091300-388-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskUci3DU/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781966829091300-388-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:37167
--webserver_interface=127.0.97.62
--webserver_port=0
--builtin_ntp_servers=127.0.97.20:45609
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.97.62:37167
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc_authentication=disabled
--rpc_encryption=disabled with env {}
W20260620 14:47:49.204846 5025 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 14:47:49.205156 5025 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 14:47:49.205241 5025 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 14:47:49.212507 5025 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 14:47:49.212618 5025 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 14:47:49.212656 5025 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 14:47:49.212704 5025 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 14:47:49.212740 5025 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 14:47:49.222046 5025 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:45609
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskUci3DU/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781966829091300-388-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskUci3DU/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781966829091300-388-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.97.62:37167
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:37167
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskUci3DU/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781966829091300-388-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.97.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskUci3DU/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781966829091300-388-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 13:41:20 UTC on ccycloud.quasar-vdagoi.root.comops.site
build id 4457462
ASAN enabled
I20260620 14:47:49.223346 5025 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 14:47:49.225124 5025 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 14:47:49.237149 5031 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 14:47:49.237233 5033 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 14:47:49.237190 5030 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 14:47:49.237958 5025 server_base.cc:1034] running on GCE node
I20260620 14:47:49.238778 5025 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 14:47:49.240411 5025 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 14:47:49.241665 5025 hybrid_clock.cc:648] HybridClock initialized: now 1781966869241626 us; error 39 us; skew 500 ppm
I20260620 14:47:49.242045 5025 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260620 14:47:49.244544 5025 webserver.cc:460] Webserver started at http://127.0.97.62:44543/ using document root <none> and password file <none>
I20260620 14:47:49.245317 5025 fs_manager.cc:359] Metadata directory not provided
I20260620 14:47:49.245460 5025 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 14:47:49.245810 5025 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 14:47:49.249015 5025 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskUci3DU/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781966829091300-388-0/minicluster-data/master-0/data/instance:
uuid: "eab9fc0bea2147b995dd1d8b3db3a5eb"
format_stamp: "Formatted at 2026-06-20 14:47:49 on dist-test-slave-q9j6"
server_key: "effcf5947b8abe3f8f9eb6e516e3f6b9"
server_key_iv: "a6bf60bc70c056d9ca650decda3dd794"
server_key_version: "encryptionkey@0"
I20260620 14:47:49.249878 5025 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskUci3DU/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781966829091300-388-0/minicluster-data/master-0/wal/instance:
uuid: "eab9fc0bea2147b995dd1d8b3db3a5eb"
format_stamp: "Formatted at 2026-06-20 14:47:49 on dist-test-slave-q9j6"
server_key: "effcf5947b8abe3f8f9eb6e516e3f6b9"
server_key_iv: "a6bf60bc70c056d9ca650decda3dd794"
server_key_version: "encryptionkey@0"
I20260620 14:47:49.255393 5025 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.000s
I20260620 14:47:49.259099 5039 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 14:47:49.260628 5025 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260620 14:47:49.260835 5025 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskUci3DU/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781966829091300-388-0/minicluster-data/master-0/data,/tmp/dist-test-taskUci3DU/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781966829091300-388-0/minicluster-data/master-0/wal
uuid: "eab9fc0bea2147b995dd1d8b3db3a5eb"
format_stamp: "Formatted at 2026-06-20 14:47:49 on dist-test-slave-q9j6"
server_key: "effcf5947b8abe3f8f9eb6e516e3f6b9"
server_key_iv: "a6bf60bc70c056d9ca650decda3dd794"
server_key_version: "encryptionkey@0"
I20260620 14:47:49.261086 5025 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskUci3DU/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781966829091300-388-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskUci3DU/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781966829091300-388-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskUci3DU/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781966829091300-388-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 14:47:49.267467 5025 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 14:47:49.268369 5025 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 14:47:49.268718 5025 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 14:47:49.287228 5025 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.62:37167
I20260620 14:47:49.287240 5090 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.62:37167 every 8 connection(s)
I20260620 14:47:49.288832 5025 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskUci3DU/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781966829091300-388-0/minicluster-data/master-0/data/info.pb
I20260620 14:47:49.292882 5091 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 14:47:49.292896 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskUci3DU/build/asan/bin/kudu as pid 5025
I20260620 14:47:49.293111 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskUci3DU/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781966829091300-388-0/minicluster-data/master-0/wal/instance
I20260620 14:47:49.293732 388 external_mini_cluster.cc:1428] Setting key c5d6dfbe51a09415a5b49ccf3cc9dc93
I20260620 14:47:49.303781 5094 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:47:49.295885 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35352 (local address 127.0.97.62:37167)
0620 14:47:49.297025 (+ 1140us) server_negotiation.cc:206] Beginning negotiation
0620 14:47:49.297044 (+ 19us) server_negotiation.cc:399] Waiting for connection header
0620 14:47:49.297096 (+ 52us) server_negotiation.cc:407] Connection header received
0620 14:47:49.299509 (+ 2413us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:47:49.299538 (+ 29us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:47:49.300179 (+ 641us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:47:49.300672 (+ 493us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:47:49.301682 (+ 1010us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 14:47:49.301689 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 14:47:49.301706 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 14:47:49.301722 (+ 16us) server_negotiation.cc:922] calling sasl_server_start()
0620 14:47:49.301797 (+ 75us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 14:47:49.301818 (+ 21us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 14:47:49.302051 (+ 233us) server_negotiation.cc:1035] Waiting for connection context
0620 14:47:49.302182 (+ 131us) server_negotiation.cc:299] Negotiation successful
0620 14:47:49.302495 (+ 313us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":569,"thread_start_us":351,"threads_started":1}
I20260620 14:47:49.307739 5091 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P eab9fc0bea2147b995dd1d8b3db3a5eb: Bootstrap starting.
I20260620 14:47:49.310698 5091 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P eab9fc0bea2147b995dd1d8b3db3a5eb: Neither blocks nor log segments found. Creating new log.
I20260620 14:47:49.311765 5091 log.cc:824] T 00000000000000000000000000000000 P eab9fc0bea2147b995dd1d8b3db3a5eb: Log is configured to *not* fsync() on all Append() calls
I20260620 14:47:49.314854 5091 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P eab9fc0bea2147b995dd1d8b3db3a5eb: No bootstrap required, opened a new log
I20260620 14:47:49.322443 5091 raft_consensus.cc:348] T 00000000000000000000000000000000 P eab9fc0bea2147b995dd1d8b3db3a5eb [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "eab9fc0bea2147b995dd1d8b3db3a5eb" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 37167 } }
I20260620 14:47:49.322820 5091 raft_consensus.cc:374] T 00000000000000000000000000000000 P eab9fc0bea2147b995dd1d8b3db3a5eb [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 14:47:49.322885 5091 raft_consensus.cc:729] T 00000000000000000000000000000000 P eab9fc0bea2147b995dd1d8b3db3a5eb [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: eab9fc0bea2147b995dd1d8b3db3a5eb, State: Initialized, Role: FOLLOWER
I20260620 14:47:49.323485 5091 consensus_queue.cc:260] T 00000000000000000000000000000000 P eab9fc0bea2147b995dd1d8b3db3a5eb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "eab9fc0bea2147b995dd1d8b3db3a5eb" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 37167 } }
I20260620 14:47:49.323808 5091 raft_consensus.cc:388] T 00000000000000000000000000000000 P eab9fc0bea2147b995dd1d8b3db3a5eb [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 14:47:49.323998 5091 raft_consensus.cc:482] T 00000000000000000000000000000000 P eab9fc0bea2147b995dd1d8b3db3a5eb [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 14:47:49.324211 5091 raft_consensus.cc:3037] T 00000000000000000000000000000000 P eab9fc0bea2147b995dd1d8b3db3a5eb [term 0 FOLLOWER]: Advancing to term 1
I20260620 14:47:49.326406 5091 raft_consensus.cc:504] T 00000000000000000000000000000000 P eab9fc0bea2147b995dd1d8b3db3a5eb [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "eab9fc0bea2147b995dd1d8b3db3a5eb" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 37167 } }
I20260620 14:47:49.326875 5091 leader_election.cc:302] T 00000000000000000000000000000000 P eab9fc0bea2147b995dd1d8b3db3a5eb [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: eab9fc0bea2147b995dd1d8b3db3a5eb; no voters:
I20260620 14:47:49.327389 5091 leader_election.cc:288] T 00000000000000000000000000000000 P eab9fc0bea2147b995dd1d8b3db3a5eb [CANDIDATE]: Term 1 election: Requested vote from peers
I20260620 14:47:49.327556 5096 raft_consensus.cc:2781] T 00000000000000000000000000000000 P eab9fc0bea2147b995dd1d8b3db3a5eb [term 1 FOLLOWER]: Leader election won for term 1
I20260620 14:47:49.328430 5096 raft_consensus.cc:686] T 00000000000000000000000000000000 P eab9fc0bea2147b995dd1d8b3db3a5eb [term 1 LEADER]: Becoming Leader. State: Replica: eab9fc0bea2147b995dd1d8b3db3a5eb, State: Running, Role: LEADER
I20260620 14:47:49.329093 5096 consensus_queue.cc:237] T 00000000000000000000000000000000 P eab9fc0bea2147b995dd1d8b3db3a5eb [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "eab9fc0bea2147b995dd1d8b3db3a5eb" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 37167 } }
I20260620 14:47:49.329706 5091 sys_catalog.cc:564] T 00000000000000000000000000000000 P eab9fc0bea2147b995dd1d8b3db3a5eb [sys.catalog]: configured and running, proceeding with master startup.
I20260620 14:47:49.331750 5098 sys_catalog.cc:455] T 00000000000000000000000000000000 P eab9fc0bea2147b995dd1d8b3db3a5eb [sys.catalog]: SysCatalogTable state changed. Reason: New leader eab9fc0bea2147b995dd1d8b3db3a5eb. Latest consensus state: current_term: 1 leader_uuid: "eab9fc0bea2147b995dd1d8b3db3a5eb" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "eab9fc0bea2147b995dd1d8b3db3a5eb" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 37167 } } }
I20260620 14:47:49.332118 5098 sys_catalog.cc:458] T 00000000000000000000000000000000 P eab9fc0bea2147b995dd1d8b3db3a5eb [sys.catalog]: This master's current role is: LEADER
I20260620 14:47:49.332947 5097 sys_catalog.cc:455] T 00000000000000000000000000000000 P eab9fc0bea2147b995dd1d8b3db3a5eb [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "eab9fc0bea2147b995dd1d8b3db3a5eb" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "eab9fc0bea2147b995dd1d8b3db3a5eb" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 37167 } } }
I20260620 14:47:49.333156 5104 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 14:47:49.333303 5097 sys_catalog.cc:458] T 00000000000000000000000000000000 P eab9fc0bea2147b995dd1d8b3db3a5eb [sys.catalog]: This master's current role is: LEADER
I20260620 14:47:49.339268 5104 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 14:47:49.349259 5104 catalog_manager.cc:1347] Generated new cluster ID: 04b3ef2f7253484b94df97042403c780
I20260620 14:47:49.349432 5104 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 14:47:49.362864 5104 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 14:47:49.363380 5104 catalog_manager.cc:1504] Loading token signing keys...
I20260620 14:47:49.370618 5104 catalog_manager.cc:5868] T 00000000000000000000000000000000 P eab9fc0bea2147b995dd1d8b3db3a5eb: Generated new TSK 0
I20260620 14:47:49.371706 5104 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260620 14:47:49.377449 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskUci3DU/build/asan/bin/kudu
/tmp/dist-test-taskUci3DU/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskUci3DU/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781966829091300-388-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskUci3DU/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781966829091300-388-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskUci3DU/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781966829091300-388-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskUci3DU/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781966829091300-388-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.1:0
--local_ip_for_outbound_sockets=127.0.97.1
--webserver_interface=127.0.97.1
--webserver_port=0
--tserver_master_addrs=127.0.97.62:37167
--builtin_ntp_servers=127.0.97.20:45609
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled
--rpc_encryption=disabled with env {}
W20260620 14:47:49.709391 5115 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 14:47:49.709726 5115 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 14:47:49.709897 5115 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 14:47:49.717410 5115 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 14:47:49.717559 5115 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 14:47:49.717731 5115 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.1
I20260620 14:47:49.727240 5115 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:45609
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskUci3DU/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781966829091300-388-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskUci3DU/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781966829091300-388-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskUci3DU/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781966829091300-388-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.97.1
--webserver_port=0
--tserver_master_addrs=127.0.97.62:37167
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.1
--log_dir=/tmp/dist-test-taskUci3DU/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781966829091300-388-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 13:41:20 UTC on ccycloud.quasar-vdagoi.root.comops.site
build id 4457462
ASAN enabled
I20260620 14:47:49.728679 5115 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 14:47:49.730755 5115 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 14:47:49.744428 5120 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 14:47:49.744354 5123 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 14:47:49.744324 5121 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 14:47:49.745694 5115 server_base.cc:1034] running on GCE node
I20260620 14:47:49.746276 5115 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 14:47:49.747216 5115 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 14:47:49.748412 5115 hybrid_clock.cc:648] HybridClock initialized: now 1781966869748367 us; error 40 us; skew 500 ppm
I20260620 14:47:49.748699 5115 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260620 14:47:49.750839 5115 webserver.cc:460] Webserver started at http://127.0.97.1:35773/ using document root <none> and password file <none>
I20260620 14:47:49.751489 5115 fs_manager.cc:359] Metadata directory not provided
I20260620 14:47:49.751574 5115 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 14:47:49.751911 5115 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 14:47:49.754669 5115 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskUci3DU/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781966829091300-388-0/minicluster-data/ts-0/data/instance:
uuid: "907857c6c4bb416299ab3c558ad9bf04"
format_stamp: "Formatted at 2026-06-20 14:47:49 on dist-test-slave-q9j6"
server_key: "875d08e92de34e95750e2fb27d66a02f"
server_key_iv: "af334a545f693f4b10bc8ff3672189a0"
server_key_version: "encryptionkey@0"
I20260620 14:47:49.755414 5115 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskUci3DU/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781966829091300-388-0/minicluster-data/ts-0/wal/instance:
uuid: "907857c6c4bb416299ab3c558ad9bf04"
format_stamp: "Formatted at 2026-06-20 14:47:49 on dist-test-slave-q9j6"
server_key: "875d08e92de34e95750e2fb27d66a02f"
server_key_iv: "af334a545f693f4b10bc8ff3672189a0"
server_key_version: "encryptionkey@0"
I20260620 14:47:49.760700 5115 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.001s sys 0.003s
I20260620 14:47:49.764196 5129 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 14:47:49.765717 5115 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.002s
I20260620 14:47:49.765887 5115 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskUci3DU/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781966829091300-388-0/minicluster-data/ts-0/data,/tmp/dist-test-taskUci3DU/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781966829091300-388-0/minicluster-data/ts-0/wal
uuid: "907857c6c4bb416299ab3c558ad9bf04"
format_stamp: "Formatted at 2026-06-20 14:47:49 on dist-test-slave-q9j6"
server_key: "875d08e92de34e95750e2fb27d66a02f"
server_key_iv: "af334a545f693f4b10bc8ff3672189a0"
server_key_version: "encryptionkey@0"
I20260620 14:47:49.766137 5115 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskUci3DU/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781966829091300-388-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskUci3DU/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781966829091300-388-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskUci3DU/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781966829091300-388-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 14:47:49.772104 5115 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 14:47:49.772878 5115 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 14:47:49.773115 5115 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 14:47:49.773967 5115 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 14:47:49.775345 5115 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 14:47:49.775430 5115 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 14:47:49.775502 5115 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 14:47:49.775556 5115 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 14:47:49.808643 5115 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.1:40157
I20260620 14:47:49.808696 5241 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.1:40157 every 8 connection(s)
I20260620 14:47:49.809965 5115 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskUci3DU/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781966829091300-388-0/minicluster-data/ts-0/data/info.pb
I20260620 14:47:49.817265 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskUci3DU/build/asan/bin/kudu as pid 5115
I20260620 14:47:49.817523 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskUci3DU/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781966829091300-388-0/minicluster-data/ts-0/wal/instance
I20260620 14:47:49.818157 388 external_mini_cluster.cc:1428] Setting key ad7722c307c964bf5f240598574c8a05
I20260620 14:47:49.819120 5244 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:47:49.812632 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.1:55301 (local address 127.0.97.62:37167)
0620 14:47:49.813090 (+ 458us) server_negotiation.cc:206] Beginning negotiation
0620 14:47:49.813116 (+ 26us) server_negotiation.cc:399] Waiting for connection header
0620 14:47:49.814263 (+ 1147us) server_negotiation.cc:407] Connection header received
0620 14:47:49.816315 (+ 2052us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:47:49.816321 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:47:49.816408 (+ 87us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:47:49.816571 (+ 163us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:47:49.818271 (+ 1700us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 14:47:49.818276 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 14:47:49.818282 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 14:47:49.818305 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0620 14:47:49.818359 (+ 54us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 14:47:49.818372 (+ 13us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 14:47:49.818522 (+ 150us) server_negotiation.cc:1035] Waiting for connection context
0620 14:47:49.818720 (+ 198us) server_negotiation.cc:299] Negotiation successful
0620 14:47:49.818790 (+ 70us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":352,"thread_start_us":261,"threads_started":1}
I20260620 14:47:49.820230 5245 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:47:49.813194 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:37167 (local address 127.0.97.1:55301)
0620 14:47:49.814088 (+ 894us) negotiation.cc:107] Waiting for socket to connect
0620 14:47:49.814135 (+ 47us) client_negotiation.cc:174] Beginning negotiation
0620 14:47:49.816018 (+ 1883us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 14:47:49.816884 (+ 866us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 14:47:49.816896 (+ 12us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 14:47:49.817046 (+ 150us) client_negotiation.cc:189] Negotiated authn=SASL
0620 14:47:49.817924 (+ 878us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0620 14:47:49.817930 (+ 6us) client_negotiation.cc:656] calling sasl_client_start()
0620 14:47:49.817972 (+ 42us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0620 14:47:49.818064 (+ 92us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 14:47:49.818488 (+ 424us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 14:47:49.818500 (+ 12us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 14:47:49.818508 (+ 8us) client_negotiation.cc:769] Sending connection context
0620 14:47:49.818811 (+ 303us) client_negotiation.cc:240] Negotiation successful
0620 14:47:49.819093 (+ 282us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":421,"thread_start_us":248,"threads_started":1}
I20260620 14:47:49.821218 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskUci3DU/build/asan/bin/kudu
/tmp/dist-test-taskUci3DU/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskUci3DU/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781966829091300-388-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskUci3DU/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781966829091300-388-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskUci3DU/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781966829091300-388-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskUci3DU/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781966829091300-388-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.2:0
--local_ip_for_outbound_sockets=127.0.97.2
--webserver_interface=127.0.97.2
--webserver_port=0
--tserver_master_addrs=127.0.97.62:37167
--builtin_ntp_servers=127.0.97.20:45609
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled
--rpc_encryption=disabled with env {}
I20260620 14:47:49.821862 5242 heartbeater.cc:347] Connected to a master server at 127.0.97.62:37167
I20260620 14:47:49.822216 5242 heartbeater.cc:464] Registering TS with master...
I20260620 14:47:49.822962 5242 heartbeater.cc:511] Master 127.0.97.62:37167 requested a full tablet report, sending...
I20260620 14:47:49.825232 5056 ts_manager.cc:194] Registered new tserver with Master: 907857c6c4bb416299ab3c558ad9bf04 (127.0.97.1:40157)
W20260620 14:47:50.154485 5246 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 14:47:50.154776 5246 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 14:47:50.154863 5246 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 14:47:50.162019 5246 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 14:47:50.162122 5246 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 14:47:50.162228 5246 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.2
I20260620 14:47:50.171773 5246 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:45609
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskUci3DU/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781966829091300-388-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskUci3DU/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781966829091300-388-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskUci3DU/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781966829091300-388-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.97.2
--webserver_port=0
--tserver_master_addrs=127.0.97.62:37167
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.2
--log_dir=/tmp/dist-test-taskUci3DU/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781966829091300-388-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 13:41:20 UTC on ccycloud.quasar-vdagoi.root.comops.site
build id 4457462
ASAN enabled
I20260620 14:47:50.173050 5246 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 14:47:50.174779 5246 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 14:47:50.188432 5251 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 14:47:50.188309 5252 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 14:47:50.188326 5254 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 14:47:50.189670 5246 server_base.cc:1034] running on GCE node
I20260620 14:47:50.190336 5246 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 14:47:50.191272 5246 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 14:47:50.192485 5246 hybrid_clock.cc:648] HybridClock initialized: now 1781966870192416 us; error 66 us; skew 500 ppm
I20260620 14:47:50.192799 5246 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260620 14:47:50.195058 5246 webserver.cc:460] Webserver started at http://127.0.97.2:43153/ using document root <none> and password file <none>
I20260620 14:47:50.195842 5246 fs_manager.cc:359] Metadata directory not provided
I20260620 14:47:50.195966 5246 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 14:47:50.196282 5246 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 14:47:50.199092 5246 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskUci3DU/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781966829091300-388-0/minicluster-data/ts-1/data/instance:
uuid: "fe615ae6a4174b948f8d28e970eb94a2"
format_stamp: "Formatted at 2026-06-20 14:47:50 on dist-test-slave-q9j6"
server_key: "5afbb70f3a42c7f5eafcdb2ea03d9fe4"
server_key_iv: "55456313e0d87bf87db92e07cd2ffbbc"
server_key_version: "encryptionkey@0"
I20260620 14:47:50.199937 5246 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskUci3DU/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781966829091300-388-0/minicluster-data/ts-1/wal/instance:
uuid: "fe615ae6a4174b948f8d28e970eb94a2"
format_stamp: "Formatted at 2026-06-20 14:47:50 on dist-test-slave-q9j6"
server_key: "5afbb70f3a42c7f5eafcdb2ea03d9fe4"
server_key_iv: "55456313e0d87bf87db92e07cd2ffbbc"
server_key_version: "encryptionkey@0"
I20260620 14:47:50.205001 5246 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.004s sys 0.000s
I20260620 14:47:50.208462 5260 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 14:47:50.209810 5246 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260620 14:47:50.209991 5246 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskUci3DU/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781966829091300-388-0/minicluster-data/ts-1/data,/tmp/dist-test-taskUci3DU/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781966829091300-388-0/minicluster-data/ts-1/wal
uuid: "fe615ae6a4174b948f8d28e970eb94a2"
format_stamp: "Formatted at 2026-06-20 14:47:50 on dist-test-slave-q9j6"
server_key: "5afbb70f3a42c7f5eafcdb2ea03d9fe4"
server_key_iv: "55456313e0d87bf87db92e07cd2ffbbc"
server_key_version: "encryptionkey@0"
I20260620 14:47:50.210247 5246 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskUci3DU/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781966829091300-388-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskUci3DU/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781966829091300-388-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskUci3DU/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781966829091300-388-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 14:47:50.216084 5246 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 14:47:50.216902 5246 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 14:47:50.217181 5246 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 14:47:50.218108 5246 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 14:47:50.219497 5246 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 14:47:50.219637 5246 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 14:47:50.219779 5246 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 14:47:50.219867 5246 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 14:47:50.252502 5246 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.2:37789
I20260620 14:47:50.252554 5372 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.2:37789 every 8 connection(s)
I20260620 14:47:50.253870 5246 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskUci3DU/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781966829091300-388-0/minicluster-data/ts-1/data/info.pb
I20260620 14:47:50.261620 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskUci3DU/build/asan/bin/kudu as pid 5246
I20260620 14:47:50.261834 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskUci3DU/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781966829091300-388-0/minicluster-data/ts-1/wal/instance
I20260620 14:47:50.262285 5244 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:47:50.256490 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.2:35243 (local address 127.0.97.62:37167)
0620 14:47:50.256668 (+ 178us) server_negotiation.cc:206] Beginning negotiation
0620 14:47:50.256673 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 14:47:50.258150 (+ 1477us) server_negotiation.cc:407] Connection header received
0620 14:47:50.259924 (+ 1774us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:47:50.259929 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:47:50.259985 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:47:50.260084 (+ 99us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:47:50.261493 (+ 1409us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 14:47:50.261499 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 14:47:50.261503 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 14:47:50.261525 (+ 22us) server_negotiation.cc:922] calling sasl_server_start()
0620 14:47:50.261562 (+ 37us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 14:47:50.261580 (+ 18us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 14:47:50.261721 (+ 141us) server_negotiation.cc:1035] Waiting for connection context
0620 14:47:50.261916 (+ 195us) server_negotiation.cc:299] Negotiation successful
0620 14:47:50.261982 (+ 66us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":65}
I20260620 14:47:50.262403 388 external_mini_cluster.cc:1428] Setting key 70d19d251068eddfc0d6f1048a17b5ce
I20260620 14:47:50.263396 5375 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:47:50.257099 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:37167 (local address 127.0.97.2:35243)
0620 14:47:50.258009 (+ 910us) negotiation.cc:107] Waiting for socket to connect
0620 14:47:50.258046 (+ 37us) client_negotiation.cc:174] Beginning negotiation
0620 14:47:50.259536 (+ 1490us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 14:47:50.260307 (+ 771us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 14:47:50.260315 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 14:47:50.260418 (+ 103us) client_negotiation.cc:189] Negotiated authn=SASL
0620 14:47:50.261132 (+ 714us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0620 14:47:50.261134 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0620 14:47:50.261179 (+ 45us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0620 14:47:50.261270 (+ 91us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 14:47:50.261684 (+ 414us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 14:47:50.261691 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 14:47:50.261697 (+ 6us) client_negotiation.cc:769] Sending connection context
0620 14:47:50.262009 (+ 312us) client_negotiation.cc:240] Negotiation successful
0620 14:47:50.262229 (+ 220us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":427,"thread_start_us":253,"threads_started":1}
I20260620 14:47:50.265132 5373 heartbeater.cc:347] Connected to a master server at 127.0.97.62:37167
I20260620 14:47:50.265450 5373 heartbeater.cc:464] Registering TS with master...
I20260620 14:47:50.265437 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskUci3DU/build/asan/bin/kudu
/tmp/dist-test-taskUci3DU/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskUci3DU/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781966829091300-388-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskUci3DU/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781966829091300-388-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskUci3DU/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781966829091300-388-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskUci3DU/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781966829091300-388-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.3:0
--local_ip_for_outbound_sockets=127.0.97.3
--webserver_interface=127.0.97.3
--webserver_port=0
--tserver_master_addrs=127.0.97.62:37167
--builtin_ntp_servers=127.0.97.20:45609
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled
--rpc_encryption=disabled with env {}
I20260620 14:47:50.266111 5373 heartbeater.cc:511] Master 127.0.97.62:37167 requested a full tablet report, sending...
I20260620 14:47:50.267589 5056 ts_manager.cc:194] Registered new tserver with Master: fe615ae6a4174b948f8d28e970eb94a2 (127.0.97.2:37789)
W20260620 14:47:50.604059 5376 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 14:47:50.604377 5376 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 14:47:50.604533 5376 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 14:47:50.611773 5376 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 14:47:50.611886 5376 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 14:47:50.612073 5376 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.3
I20260620 14:47:50.627097 5376 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:45609
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskUci3DU/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781966829091300-388-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskUci3DU/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781966829091300-388-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskUci3DU/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781966829091300-388-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.97.3
--webserver_port=0
--tserver_master_addrs=127.0.97.62:37167
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.3
--log_dir=/tmp/dist-test-taskUci3DU/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781966829091300-388-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 13:41:20 UTC on ccycloud.quasar-vdagoi.root.comops.site
build id 4457462
ASAN enabled
I20260620 14:47:50.628532 5376 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 14:47:50.630540 5376 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 14:47:50.643561 5381 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 14:47:50.643643 5382 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 14:47:50.643643 5384 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 14:47:50.644832 5376 server_base.cc:1034] running on GCE node
I20260620 14:47:50.645469 5376 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 14:47:50.646447 5376 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 14:47:50.647702 5376 hybrid_clock.cc:648] HybridClock initialized: now 1781966870647646 us; error 42 us; skew 500 ppm
I20260620 14:47:50.648030 5376 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260620 14:47:50.650396 5376 webserver.cc:460] Webserver started at http://127.0.97.3:46087/ using document root <none> and password file <none>
I20260620 14:47:50.651140 5376 fs_manager.cc:359] Metadata directory not provided
I20260620 14:47:50.651255 5376 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 14:47:50.651587 5376 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 14:47:50.654553 5376 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskUci3DU/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781966829091300-388-0/minicluster-data/ts-2/data/instance:
uuid: "212b7843c32b4cca929902031c380a58"
format_stamp: "Formatted at 2026-06-20 14:47:50 on dist-test-slave-q9j6"
server_key: "700c7d288c79f081e4d6b83a3b9385f2"
server_key_iv: "df51cfb16ccd5c3434e5774fb7e7cd53"
server_key_version: "encryptionkey@0"
I20260620 14:47:50.655376 5376 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskUci3DU/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781966829091300-388-0/minicluster-data/ts-2/wal/instance:
uuid: "212b7843c32b4cca929902031c380a58"
format_stamp: "Formatted at 2026-06-20 14:47:50 on dist-test-slave-q9j6"
server_key: "700c7d288c79f081e4d6b83a3b9385f2"
server_key_iv: "df51cfb16ccd5c3434e5774fb7e7cd53"
server_key_version: "encryptionkey@0"
I20260620 14:47:50.660861 5376 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.001s sys 0.003s
I20260620 14:47:50.664597 5390 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 14:47:50.666402 5376 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.001s
I20260620 14:47:50.666606 5376 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskUci3DU/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781966829091300-388-0/minicluster-data/ts-2/data,/tmp/dist-test-taskUci3DU/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781966829091300-388-0/minicluster-data/ts-2/wal
uuid: "212b7843c32b4cca929902031c380a58"
format_stamp: "Formatted at 2026-06-20 14:47:50 on dist-test-slave-q9j6"
server_key: "700c7d288c79f081e4d6b83a3b9385f2"
server_key_iv: "df51cfb16ccd5c3434e5774fb7e7cd53"
server_key_version: "encryptionkey@0"
I20260620 14:47:50.666858 5376 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskUci3DU/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781966829091300-388-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskUci3DU/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781966829091300-388-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskUci3DU/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781966829091300-388-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 14:47:50.673120 5376 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 14:47:50.673991 5376 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 14:47:50.674283 5376 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 14:47:50.675298 5376 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 14:47:50.676819 5376 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 14:47:50.676944 5376 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 14:47:50.677086 5376 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 14:47:50.677181 5376 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 14:47:50.710831 5376 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.3:45123
I20260620 14:47:50.710923 5502 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.3:45123 every 8 connection(s)
I20260620 14:47:50.712280 5376 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskUci3DU/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781966829091300-388-0/minicluster-data/ts-2/data/info.pb
I20260620 14:47:50.716372 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskUci3DU/build/asan/bin/kudu as pid 5376
I20260620 14:47:50.716576 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskUci3DU/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781966829091300-388-0/minicluster-data/ts-2/wal/instance
I20260620 14:47:50.717067 388 external_mini_cluster.cc:1428] Setting key 5a265702a653daabcefc921011b9afd8
I20260620 14:47:50.721032 5244 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:47:50.714982 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.3:57859 (local address 127.0.97.62:37167)
0620 14:47:50.715191 (+ 209us) server_negotiation.cc:206] Beginning negotiation
0620 14:47:50.715196 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 14:47:50.716851 (+ 1655us) server_negotiation.cc:407] Connection header received
0620 14:47:50.718713 (+ 1862us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:47:50.718719 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:47:50.718786 (+ 67us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:47:50.718908 (+ 122us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:47:50.720307 (+ 1399us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 14:47:50.720313 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 14:47:50.720319 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 14:47:50.720337 (+ 18us) server_negotiation.cc:922] calling sasl_server_start()
0620 14:47:50.720376 (+ 39us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 14:47:50.720387 (+ 11us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 14:47:50.720517 (+ 130us) server_negotiation.cc:1035] Waiting for connection context
0620 14:47:50.720711 (+ 194us) server_negotiation.cc:299] Negotiation successful
0620 14:47:50.720778 (+ 67us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":66}
I20260620 14:47:50.722072 5505 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:47:50.715586 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:37167 (local address 127.0.97.3:57859)
0620 14:47:50.716668 (+ 1082us) negotiation.cc:107] Waiting for socket to connect
0620 14:47:50.716722 (+ 54us) client_negotiation.cc:174] Beginning negotiation
0620 14:47:50.718448 (+ 1726us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 14:47:50.719163 (+ 715us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 14:47:50.719172 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 14:47:50.719273 (+ 101us) client_negotiation.cc:189] Negotiated authn=SASL
0620 14:47:50.720017 (+ 744us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0620 14:47:50.720019 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0620 14:47:50.720058 (+ 39us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0620 14:47:50.720145 (+ 87us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 14:47:50.720488 (+ 343us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 14:47:50.720501 (+ 13us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 14:47:50.720509 (+ 8us) client_negotiation.cc:769] Sending connection context
0620 14:47:50.720797 (+ 288us) client_negotiation.cc:240] Negotiation successful
0620 14:47:50.721090 (+ 293us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":460,"thread_start_us":286,"threads_started":1}
I20260620 14:47:50.723490 5503 heartbeater.cc:347] Connected to a master server at 127.0.97.62:37167
I20260620 14:47:50.723841 5503 heartbeater.cc:464] Registering TS with master...
I20260620 14:47:50.724468 5503 heartbeater.cc:511] Master 127.0.97.62:37167 requested a full tablet report, sending...
I20260620 14:47:50.725890 5056 ts_manager.cc:194] Registered new tserver with Master: 212b7843c32b4cca929902031c380a58 (127.0.97.3:45123)
I20260620 14:47:50.732514 388 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260620 14:47:50.739198 5244 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:47:50.736863 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40902 (local address 127.0.97.62:37167)
0620 14:47:50.737095 (+ 232us) server_negotiation.cc:206] Beginning negotiation
0620 14:47:50.737104 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0620 14:47:50.737454 (+ 350us) server_negotiation.cc:407] Connection header received
0620 14:47:50.737594 (+ 140us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:47:50.737598 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:47:50.737648 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:47:50.737726 (+ 78us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:47:50.738629 (+ 903us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 14:47:50.738633 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 14:47:50.738636 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 14:47:50.738647 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0620 14:47:50.738671 (+ 24us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 14:47:50.738681 (+ 10us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 14:47:50.738769 (+ 88us) server_negotiation.cc:1035] Waiting for connection context
0620 14:47:50.738855 (+ 86us) server_negotiation.cc:299] Negotiation successful
0620 14:47:50.738898 (+ 43us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":69}
I20260620 14:47:50.743002 5056 catalog_manager.cc:1908] Servicing CreateTable request from {username='slave'} at 127.0.0.1:40902:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260620 14:47:50.749724 5056 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260620 14:47:50.768748 5516 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:47:50.762192 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57860 (local address 127.0.97.1:40157)
0620 14:47:50.763397 (+ 1205us) server_negotiation.cc:206] Beginning negotiation
0620 14:47:50.763407 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0620 14:47:50.764221 (+ 814us) server_negotiation.cc:407] Connection header received
0620 14:47:50.764526 (+ 305us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:47:50.764539 (+ 13us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:47:50.764810 (+ 271us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:47:50.764937 (+ 127us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:47:50.767501 (+ 2564us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 14:47:50.767514 (+ 13us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 14:47:50.767534 (+ 20us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 14:47:50.767567 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0620 14:47:50.767667 (+ 100us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 14:47:50.767690 (+ 23us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 14:47:50.767797 (+ 107us) server_negotiation.cc:1035] Waiting for connection context
0620 14:47:50.768261 (+ 464us) server_negotiation.cc:299] Negotiation successful
0620 14:47:50.768371 (+ 110us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1037,"thread_start_us":915,"threads_started":1}
I20260620 14:47:50.769503 5518 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:47:50.763433 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36042 (local address 127.0.97.2:37789)
0620 14:47:50.764001 (+ 568us) server_negotiation.cc:206] Beginning negotiation
0620 14:47:50.764009 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0620 14:47:50.764819 (+ 810us) server_negotiation.cc:407] Connection header received
0620 14:47:50.764895 (+ 76us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:47:50.764902 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:47:50.766305 (+ 1403us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:47:50.766458 (+ 153us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:47:50.768145 (+ 1687us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 14:47:50.768155 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 14:47:50.768169 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 14:47:50.768201 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0620 14:47:50.768257 (+ 56us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 14:47:50.768277 (+ 20us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 14:47:50.768393 (+ 116us) server_negotiation.cc:1035] Waiting for connection context
0620 14:47:50.769073 (+ 680us) server_negotiation.cc:299] Negotiation successful
0620 14:47:50.769174 (+ 101us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":396,"thread_start_us":805,"threads_started":1}
I20260620 14:47:50.770031 5515 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:47:50.762716 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.2:37789 (local address 127.0.0.1:36042)
0620 14:47:50.764574 (+ 1858us) negotiation.cc:107] Waiting for socket to connect
0620 14:47:50.764590 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0620 14:47:50.764672 (+ 82us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 14:47:50.766673 (+ 2001us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 14:47:50.766679 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 14:47:50.766702 (+ 23us) client_negotiation.cc:189] Negotiated authn=SASL
0620 14:47:50.767926 (+ 1224us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0620 14:47:50.767929 (+ 3us) client_negotiation.cc:656] calling sasl_client_start()
0620 14:47:50.767945 (+ 16us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0620 14:47:50.767992 (+ 47us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 14:47:50.768457 (+ 465us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 14:47:50.768462 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 14:47:50.768467 (+ 5us) client_negotiation.cc:769] Sending connection context
0620 14:47:50.768603 (+ 136us) client_negotiation.cc:240] Negotiation successful
0620 14:47:50.768693 (+ 90us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1715,"thread_start_us":906,"threads_started":1}
I20260620 14:47:50.770885 5517 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:47:50.761968 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:40157 (local address 127.0.0.1:57860)
0620 14:47:50.764051 (+ 2083us) negotiation.cc:107] Waiting for socket to connect
0620 14:47:50.764101 (+ 50us) client_negotiation.cc:174] Beginning negotiation
0620 14:47:50.764345 (+ 244us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 14:47:50.766057 (+ 1712us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 14:47:50.766067 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 14:47:50.766172 (+ 105us) client_negotiation.cc:189] Negotiated authn=SASL
0620 14:47:50.767161 (+ 989us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0620 14:47:50.767166 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0620 14:47:50.767199 (+ 33us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0620 14:47:50.767295 (+ 96us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 14:47:50.768040 (+ 745us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 14:47:50.768054 (+ 14us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 14:47:50.768064 (+ 10us) client_negotiation.cc:769] Sending connection context
0620 14:47:50.770279 (+ 2215us) client_negotiation.cc:240] Negotiation successful
0620 14:47:50.770330 (+ 51us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1801,"thread_start_us":1635,"threads_started":1}
I20260620 14:47:50.773954 5515 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:47:50.764485 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.3:45123 (local address 127.0.0.1:33850)
0620 14:47:50.770362 (+ 5877us) negotiation.cc:107] Waiting for socket to connect
0620 14:47:50.770380 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0620 14:47:50.770472 (+ 92us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 14:47:50.772110 (+ 1638us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 14:47:50.772116 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 14:47:50.772150 (+ 34us) client_negotiation.cc:189] Negotiated authn=SASL
0620 14:47:50.772805 (+ 655us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0620 14:47:50.772810 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0620 14:47:50.772829 (+ 19us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0620 14:47:50.772878 (+ 49us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 14:47:50.773522 (+ 644us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 14:47:50.773528 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 14:47:50.773534 (+ 6us) client_negotiation.cc:769] Sending connection context
0620 14:47:50.773616 (+ 82us) client_negotiation.cc:240] Negotiation successful
0620 14:47:50.773673 (+ 57us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":5781,"thread_start_us":6672,"threads_started":1}
I20260620 14:47:50.774876 5520 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:47:50.764966 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33850 (local address 127.0.97.3:45123)
0620 14:47:50.771470 (+ 6504us) server_negotiation.cc:206] Beginning negotiation
0620 14:47:50.771487 (+ 17us) server_negotiation.cc:399] Waiting for connection header
0620 14:47:50.771522 (+ 35us) server_negotiation.cc:407] Connection header received
0620 14:47:50.771588 (+ 66us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:47:50.771618 (+ 30us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:47:50.771878 (+ 260us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:47:50.772129 (+ 251us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:47:50.773262 (+ 1133us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 14:47:50.773271 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 14:47:50.773286 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 14:47:50.773306 (+ 20us) server_negotiation.cc:922] calling sasl_server_start()
0620 14:47:50.773358 (+ 52us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 14:47:50.773378 (+ 20us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 14:47:50.774398 (+ 1020us) server_negotiation.cc:1035] Waiting for connection context
0620 14:47:50.774482 (+ 84us) server_negotiation.cc:299] Negotiation successful
0620 14:47:50.774579 (+ 97us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":6312,"thread_start_us":6459,"threads_started":1}
I20260620 14:47:50.774801 5308 tablet_service.cc:1467] Processing CreateTablet for tablet 93bee0eb11a44bd399865099b5b51f23 (DEFAULT_TABLE table=test-table [id=79204da38ca14f569d18f1e22107c116]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 14:47:50.774801 5177 tablet_service.cc:1467] Processing CreateTablet for tablet 93bee0eb11a44bd399865099b5b51f23 (DEFAULT_TABLE table=test-table [id=79204da38ca14f569d18f1e22107c116]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 14:47:50.776353 5177 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 93bee0eb11a44bd399865099b5b51f23. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 14:47:50.776356 5308 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 93bee0eb11a44bd399865099b5b51f23. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 14:47:50.777978 5438 tablet_service.cc:1467] Processing CreateTablet for tablet 93bee0eb11a44bd399865099b5b51f23 (DEFAULT_TABLE table=test-table [id=79204da38ca14f569d18f1e22107c116]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 14:47:50.779356 5438 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 93bee0eb11a44bd399865099b5b51f23. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 14:47:50.793565 5522 tablet_bootstrap.cc:492] T 93bee0eb11a44bd399865099b5b51f23 P fe615ae6a4174b948f8d28e970eb94a2: Bootstrap starting.
I20260620 14:47:50.794526 5521 tablet_bootstrap.cc:492] T 93bee0eb11a44bd399865099b5b51f23 P 212b7843c32b4cca929902031c380a58: Bootstrap starting.
I20260620 14:47:50.795917 5523 tablet_bootstrap.cc:492] T 93bee0eb11a44bd399865099b5b51f23 P 907857c6c4bb416299ab3c558ad9bf04: Bootstrap starting.
I20260620 14:47:50.797744 5522 tablet_bootstrap.cc:654] T 93bee0eb11a44bd399865099b5b51f23 P fe615ae6a4174b948f8d28e970eb94a2: Neither blocks nor log segments found. Creating new log.
I20260620 14:47:50.797797 5521 tablet_bootstrap.cc:654] T 93bee0eb11a44bd399865099b5b51f23 P 212b7843c32b4cca929902031c380a58: Neither blocks nor log segments found. Creating new log.
I20260620 14:47:50.798815 5521 log.cc:824] T 93bee0eb11a44bd399865099b5b51f23 P 212b7843c32b4cca929902031c380a58: Log is configured to *not* fsync() on all Append() calls
I20260620 14:47:50.798964 5522 log.cc:824] T 93bee0eb11a44bd399865099b5b51f23 P fe615ae6a4174b948f8d28e970eb94a2: Log is configured to *not* fsync() on all Append() calls
I20260620 14:47:50.799335 5523 tablet_bootstrap.cc:654] T 93bee0eb11a44bd399865099b5b51f23 P 907857c6c4bb416299ab3c558ad9bf04: Neither blocks nor log segments found. Creating new log.
I20260620 14:47:50.800556 5523 log.cc:824] T 93bee0eb11a44bd399865099b5b51f23 P 907857c6c4bb416299ab3c558ad9bf04: Log is configured to *not* fsync() on all Append() calls
I20260620 14:47:50.802371 5522 tablet_bootstrap.cc:492] T 93bee0eb11a44bd399865099b5b51f23 P fe615ae6a4174b948f8d28e970eb94a2: No bootstrap required, opened a new log
I20260620 14:47:50.802373 5521 tablet_bootstrap.cc:492] T 93bee0eb11a44bd399865099b5b51f23 P 212b7843c32b4cca929902031c380a58: No bootstrap required, opened a new log
I20260620 14:47:50.802675 5521 ts_tablet_manager.cc:1397] T 93bee0eb11a44bd399865099b5b51f23 P 212b7843c32b4cca929902031c380a58: Time spent bootstrapping tablet: real 0.008s user 0.003s sys 0.003s
I20260620 14:47:50.802683 5522 ts_tablet_manager.cc:1397] T 93bee0eb11a44bd399865099b5b51f23 P fe615ae6a4174b948f8d28e970eb94a2: Time spent bootstrapping tablet: real 0.011s user 0.004s sys 0.002s
I20260620 14:47:50.803442 5523 tablet_bootstrap.cc:492] T 93bee0eb11a44bd399865099b5b51f23 P 907857c6c4bb416299ab3c558ad9bf04: No bootstrap required, opened a new log
I20260620 14:47:50.803725 5523 ts_tablet_manager.cc:1397] T 93bee0eb11a44bd399865099b5b51f23 P 907857c6c4bb416299ab3c558ad9bf04: Time spent bootstrapping tablet: real 0.008s user 0.007s sys 0.000s
I20260620 14:47:50.810506 5521 raft_consensus.cc:348] T 93bee0eb11a44bd399865099b5b51f23 P 212b7843c32b4cca929902031c380a58 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "907857c6c4bb416299ab3c558ad9bf04" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 40157 } } peers { permanent_uuid: "212b7843c32b4cca929902031c380a58" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 45123 } } peers { permanent_uuid: "fe615ae6a4174b948f8d28e970eb94a2" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 37789 } }
I20260620 14:47:50.810936 5521 raft_consensus.cc:374] T 93bee0eb11a44bd399865099b5b51f23 P 212b7843c32b4cca929902031c380a58 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 14:47:50.811033 5521 raft_consensus.cc:729] T 93bee0eb11a44bd399865099b5b51f23 P 212b7843c32b4cca929902031c380a58 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 212b7843c32b4cca929902031c380a58, State: Initialized, Role: FOLLOWER
I20260620 14:47:50.811743 5521 consensus_queue.cc:260] T 93bee0eb11a44bd399865099b5b51f23 P 212b7843c32b4cca929902031c380a58 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "907857c6c4bb416299ab3c558ad9bf04" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 40157 } } peers { permanent_uuid: "212b7843c32b4cca929902031c380a58" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 45123 } } peers { permanent_uuid: "fe615ae6a4174b948f8d28e970eb94a2" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 37789 } }
I20260620 14:47:50.813722 5522 raft_consensus.cc:348] T 93bee0eb11a44bd399865099b5b51f23 P fe615ae6a4174b948f8d28e970eb94a2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "907857c6c4bb416299ab3c558ad9bf04" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 40157 } } peers { permanent_uuid: "212b7843c32b4cca929902031c380a58" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 45123 } } peers { permanent_uuid: "fe615ae6a4174b948f8d28e970eb94a2" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 37789 } }
I20260620 14:47:50.814281 5522 raft_consensus.cc:374] T 93bee0eb11a44bd399865099b5b51f23 P fe615ae6a4174b948f8d28e970eb94a2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 14:47:50.814384 5522 raft_consensus.cc:729] T 93bee0eb11a44bd399865099b5b51f23 P fe615ae6a4174b948f8d28e970eb94a2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fe615ae6a4174b948f8d28e970eb94a2, State: Initialized, Role: FOLLOWER
I20260620 14:47:50.814814 5503 heartbeater.cc:503] Master 127.0.97.62:37167 was elected leader, sending a full tablet report...
I20260620 14:47:50.815160 5522 consensus_queue.cc:260] T 93bee0eb11a44bd399865099b5b51f23 P fe615ae6a4174b948f8d28e970eb94a2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "907857c6c4bb416299ab3c558ad9bf04" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 40157 } } peers { permanent_uuid: "212b7843c32b4cca929902031c380a58" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 45123 } } peers { permanent_uuid: "fe615ae6a4174b948f8d28e970eb94a2" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 37789 } }
I20260620 14:47:50.815372 5521 ts_tablet_manager.cc:1428] T 93bee0eb11a44bd399865099b5b51f23 P 212b7843c32b4cca929902031c380a58: Time spent starting tablet: real 0.013s user 0.012s sys 0.000s
I20260620 14:47:50.817054 5522 ts_tablet_manager.cc:1428] T 93bee0eb11a44bd399865099b5b51f23 P fe615ae6a4174b948f8d28e970eb94a2: Time spent starting tablet: real 0.014s user 0.009s sys 0.006s
I20260620 14:47:50.816893 5523 raft_consensus.cc:348] T 93bee0eb11a44bd399865099b5b51f23 P 907857c6c4bb416299ab3c558ad9bf04 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "907857c6c4bb416299ab3c558ad9bf04" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 40157 } } peers { permanent_uuid: "212b7843c32b4cca929902031c380a58" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 45123 } } peers { permanent_uuid: "fe615ae6a4174b948f8d28e970eb94a2" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 37789 } }
I20260620 14:47:50.817346 5523 raft_consensus.cc:374] T 93bee0eb11a44bd399865099b5b51f23 P 907857c6c4bb416299ab3c558ad9bf04 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 14:47:50.817479 5523 raft_consensus.cc:729] T 93bee0eb11a44bd399865099b5b51f23 P 907857c6c4bb416299ab3c558ad9bf04 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 907857c6c4bb416299ab3c558ad9bf04, State: Initialized, Role: FOLLOWER
I20260620 14:47:50.817849 5373 heartbeater.cc:503] Master 127.0.97.62:37167 was elected leader, sending a full tablet report...
I20260620 14:47:50.818120 5523 consensus_queue.cc:260] T 93bee0eb11a44bd399865099b5b51f23 P 907857c6c4bb416299ab3c558ad9bf04 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "907857c6c4bb416299ab3c558ad9bf04" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 40157 } } peers { permanent_uuid: "212b7843c32b4cca929902031c380a58" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 45123 } } peers { permanent_uuid: "fe615ae6a4174b948f8d28e970eb94a2" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 37789 } }
I20260620 14:47:50.822942 5242 heartbeater.cc:503] Master 127.0.97.62:37167 was elected leader, sending a full tablet report...
I20260620 14:47:50.823678 5523 ts_tablet_manager.cc:1428] T 93bee0eb11a44bd399865099b5b51f23 P 907857c6c4bb416299ab3c558ad9bf04: Time spent starting tablet: real 0.020s user 0.018s sys 0.000s
W20260620 14:47:50.964061 5504 tablet.cc:2369] T 93bee0eb11a44bd399865099b5b51f23 P 212b7843c32b4cca929902031c380a58: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260620 14:47:51.006150 5374 tablet.cc:2369] T 93bee0eb11a44bd399865099b5b51f23 P fe615ae6a4174b948f8d28e970eb94a2: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260620 14:47:51.062557 5243 tablet.cc:2369] T 93bee0eb11a44bd399865099b5b51f23 P 907857c6c4bb416299ab3c558ad9bf04: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260620 14:47:51.177062 5528 raft_consensus.cc:482] T 93bee0eb11a44bd399865099b5b51f23 P fe615ae6a4174b948f8d28e970eb94a2 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260620 14:47:51.177368 5528 raft_consensus.cc:504] T 93bee0eb11a44bd399865099b5b51f23 P fe615ae6a4174b948f8d28e970eb94a2 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "907857c6c4bb416299ab3c558ad9bf04" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 40157 } } peers { permanent_uuid: "212b7843c32b4cca929902031c380a58" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 45123 } } peers { permanent_uuid: "fe615ae6a4174b948f8d28e970eb94a2" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 37789 } }
I20260620 14:47:51.179039 5528 leader_election.cc:288] T 93bee0eb11a44bd399865099b5b51f23 P fe615ae6a4174b948f8d28e970eb94a2 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 907857c6c4bb416299ab3c558ad9bf04 (127.0.97.1:40157), 212b7843c32b4cca929902031c380a58 (127.0.97.3:45123)
I20260620 14:47:51.182448 5516 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:47:51.179400 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.2:47233 (local address 127.0.97.1:40157)
0620 14:47:51.179669 (+ 269us) server_negotiation.cc:206] Beginning negotiation
0620 14:47:51.179679 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0620 14:47:51.180125 (+ 446us) server_negotiation.cc:407] Connection header received
0620 14:47:51.180332 (+ 207us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:47:51.180338 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:47:51.180401 (+ 63us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:47:51.180485 (+ 84us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:47:51.181772 (+ 1287us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 14:47:51.181777 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 14:47:51.181789 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 14:47:51.181806 (+ 17us) server_negotiation.cc:922] calling sasl_server_start()
0620 14:47:51.181834 (+ 28us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 14:47:51.181846 (+ 12us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 14:47:51.181994 (+ 148us) server_negotiation.cc:1035] Waiting for connection context
0620 14:47:51.182049 (+ 55us) server_negotiation.cc:299] Negotiation successful
0620 14:47:51.182113 (+ 64us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":70}
I20260620 14:47:51.182595 5520 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:47:51.180219 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.2:34155 (local address 127.0.97.3:45123)
0620 14:47:51.180386 (+ 167us) server_negotiation.cc:206] Beginning negotiation
0620 14:47:51.180433 (+ 47us) server_negotiation.cc:399] Waiting for connection header
0620 14:47:51.180766 (+ 333us) server_negotiation.cc:407] Connection header received
0620 14:47:51.180868 (+ 102us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:47:51.180875 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:47:51.180936 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:47:51.181046 (+ 110us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:47:51.181966 (+ 920us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 14:47:51.181972 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 14:47:51.181976 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 14:47:51.181994 (+ 18us) server_negotiation.cc:922] calling sasl_server_start()
0620 14:47:51.182030 (+ 36us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 14:47:51.182044 (+ 14us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 14:47:51.182160 (+ 116us) server_negotiation.cc:1035] Waiting for connection context
0620 14:47:51.182289 (+ 129us) server_negotiation.cc:299] Negotiation successful
0620 14:47:51.182343 (+ 54us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":64}
I20260620 14:47:51.182655 5530 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:47:51.179324 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:40157 (local address 127.0.97.2:47233)
0620 14:47:51.179974 (+ 650us) negotiation.cc:107] Waiting for socket to connect
0620 14:47:51.179997 (+ 23us) client_negotiation.cc:174] Beginning negotiation
0620 14:47:51.180152 (+ 155us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 14:47:51.180668 (+ 516us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 14:47:51.180671 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 14:47:51.180690 (+ 19us) client_negotiation.cc:189] Negotiated authn=SASL
0620 14:47:51.181551 (+ 861us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0620 14:47:51.181554 (+ 3us) client_negotiation.cc:656] calling sasl_client_start()
0620 14:47:51.181568 (+ 14us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0620 14:47:51.181606 (+ 38us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 14:47:51.181955 (+ 349us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 14:47:51.181958 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 14:47:51.181961 (+ 3us) client_negotiation.cc:769] Sending connection context
0620 14:47:51.182062 (+ 101us) client_negotiation.cc:240] Negotiation successful
0620 14:47:51.182139 (+ 77us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":516,"thread_start_us":358,"threads_started":1}
I20260620 14:47:51.182931 5531 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:47:51.180028 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.3:45123 (local address 127.0.97.2:34155)
0620 14:47:51.180651 (+ 623us) negotiation.cc:107] Waiting for socket to connect
0620 14:47:51.180665 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0620 14:47:51.180767 (+ 102us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 14:47:51.181071 (+ 304us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 14:47:51.181074 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 14:47:51.181091 (+ 17us) client_negotiation.cc:189] Negotiated authn=SASL
0620 14:47:51.181766 (+ 675us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0620 14:47:51.181769 (+ 3us) client_negotiation.cc:656] calling sasl_client_start()
0620 14:47:51.181782 (+ 13us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0620 14:47:51.181814 (+ 32us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 14:47:51.182155 (+ 341us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 14:47:51.182160 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 14:47:51.182165 (+ 5us) client_negotiation.cc:769] Sending connection context
0620 14:47:51.182310 (+ 145us) client_negotiation.cc:240] Negotiation successful
0620 14:47:51.182353 (+ 43us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":525,"thread_start_us":320,"threads_started":1}
I20260620 14:47:51.183734 5197 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "93bee0eb11a44bd399865099b5b51f23" candidate_uuid: "fe615ae6a4174b948f8d28e970eb94a2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "907857c6c4bb416299ab3c558ad9bf04" is_pre_election: true
I20260620 14:47:51.183837 5458 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "93bee0eb11a44bd399865099b5b51f23" candidate_uuid: "fe615ae6a4174b948f8d28e970eb94a2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "212b7843c32b4cca929902031c380a58" is_pre_election: true
I20260620 14:47:51.184278 5458 raft_consensus.cc:2454] T 93bee0eb11a44bd399865099b5b51f23 P 212b7843c32b4cca929902031c380a58 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate fe615ae6a4174b948f8d28e970eb94a2 in term 0.
I20260620 14:47:51.184278 5197 raft_consensus.cc:2454] T 93bee0eb11a44bd399865099b5b51f23 P 907857c6c4bb416299ab3c558ad9bf04 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate fe615ae6a4174b948f8d28e970eb94a2 in term 0.
I20260620 14:47:51.184906 5261 leader_election.cc:302] T 93bee0eb11a44bd399865099b5b51f23 P fe615ae6a4174b948f8d28e970eb94a2 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 212b7843c32b4cca929902031c380a58, fe615ae6a4174b948f8d28e970eb94a2; no voters:
I20260620 14:47:51.185298 5528 raft_consensus.cc:2781] T 93bee0eb11a44bd399865099b5b51f23 P fe615ae6a4174b948f8d28e970eb94a2 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260620 14:47:51.185518 5528 raft_consensus.cc:482] T 93bee0eb11a44bd399865099b5b51f23 P fe615ae6a4174b948f8d28e970eb94a2 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260620 14:47:51.185635 5528 raft_consensus.cc:3037] T 93bee0eb11a44bd399865099b5b51f23 P fe615ae6a4174b948f8d28e970eb94a2 [term 0 FOLLOWER]: Advancing to term 1
I20260620 14:47:51.187948 5528 raft_consensus.cc:504] T 93bee0eb11a44bd399865099b5b51f23 P fe615ae6a4174b948f8d28e970eb94a2 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "907857c6c4bb416299ab3c558ad9bf04" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 40157 } } peers { permanent_uuid: "212b7843c32b4cca929902031c380a58" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 45123 } } peers { permanent_uuid: "fe615ae6a4174b948f8d28e970eb94a2" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 37789 } }
I20260620 14:47:51.188534 5528 leader_election.cc:288] T 93bee0eb11a44bd399865099b5b51f23 P fe615ae6a4174b948f8d28e970eb94a2 [CANDIDATE]: Term 1 election: Requested vote from peers 907857c6c4bb416299ab3c558ad9bf04 (127.0.97.1:40157), 212b7843c32b4cca929902031c380a58 (127.0.97.3:45123)
I20260620 14:47:51.189059 5197 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "93bee0eb11a44bd399865099b5b51f23" candidate_uuid: "fe615ae6a4174b948f8d28e970eb94a2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "907857c6c4bb416299ab3c558ad9bf04"
I20260620 14:47:51.189186 5458 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "93bee0eb11a44bd399865099b5b51f23" candidate_uuid: "fe615ae6a4174b948f8d28e970eb94a2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "212b7843c32b4cca929902031c380a58"
I20260620 14:47:51.189267 5197 raft_consensus.cc:3037] T 93bee0eb11a44bd399865099b5b51f23 P 907857c6c4bb416299ab3c558ad9bf04 [term 0 FOLLOWER]: Advancing to term 1
I20260620 14:47:51.189419 5458 raft_consensus.cc:3037] T 93bee0eb11a44bd399865099b5b51f23 P 212b7843c32b4cca929902031c380a58 [term 0 FOLLOWER]: Advancing to term 1
I20260620 14:47:51.191535 5197 raft_consensus.cc:2454] T 93bee0eb11a44bd399865099b5b51f23 P 907857c6c4bb416299ab3c558ad9bf04 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate fe615ae6a4174b948f8d28e970eb94a2 in term 1.
I20260620 14:47:51.191663 5458 raft_consensus.cc:2454] T 93bee0eb11a44bd399865099b5b51f23 P 212b7843c32b4cca929902031c380a58 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate fe615ae6a4174b948f8d28e970eb94a2 in term 1.
I20260620 14:47:51.192052 5261 leader_election.cc:302] T 93bee0eb11a44bd399865099b5b51f23 P fe615ae6a4174b948f8d28e970eb94a2 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 907857c6c4bb416299ab3c558ad9bf04, fe615ae6a4174b948f8d28e970eb94a2; no voters:
I20260620 14:47:51.192370 5528 raft_consensus.cc:2781] T 93bee0eb11a44bd399865099b5b51f23 P fe615ae6a4174b948f8d28e970eb94a2 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 14:47:51.192873 5528 raft_consensus.cc:686] T 93bee0eb11a44bd399865099b5b51f23 P fe615ae6a4174b948f8d28e970eb94a2 [term 1 LEADER]: Becoming Leader. State: Replica: fe615ae6a4174b948f8d28e970eb94a2, State: Running, Role: LEADER
I20260620 14:47:51.193341 5528 consensus_queue.cc:237] T 93bee0eb11a44bd399865099b5b51f23 P fe615ae6a4174b948f8d28e970eb94a2 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "907857c6c4bb416299ab3c558ad9bf04" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 40157 } } peers { permanent_uuid: "212b7843c32b4cca929902031c380a58" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 45123 } } peers { permanent_uuid: "fe615ae6a4174b948f8d28e970eb94a2" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 37789 } }
I20260620 14:47:51.198276 5055 catalog_manager.cc:5495] T 93bee0eb11a44bd399865099b5b51f23 P fe615ae6a4174b948f8d28e970eb94a2 reported cstate change: term changed from 0 to 1, leader changed from <none> to fe615ae6a4174b948f8d28e970eb94a2 (127.0.97.2). New cstate: current_term: 1 leader_uuid: "fe615ae6a4174b948f8d28e970eb94a2" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "907857c6c4bb416299ab3c558ad9bf04" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 40157 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "212b7843c32b4cca929902031c380a58" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 45123 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "fe615ae6a4174b948f8d28e970eb94a2" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 37789 } health_report { overall_health: HEALTHY } } }
I20260620 14:47:51.226097 5518 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:47:51.223838 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36048 (local address 127.0.97.2:37789)
0620 14:47:51.224060 (+ 222us) server_negotiation.cc:206] Beginning negotiation
0620 14:47:51.224067 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0620 14:47:51.224148 (+ 81us) server_negotiation.cc:407] Connection header received
0620 14:47:51.224390 (+ 242us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:47:51.224394 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:47:51.224472 (+ 78us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:47:51.224582 (+ 110us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:47:51.225464 (+ 882us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 14:47:51.225470 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 14:47:51.225475 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 14:47:51.225497 (+ 22us) server_negotiation.cc:922] calling sasl_server_start()
0620 14:47:51.225525 (+ 28us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 14:47:51.225536 (+ 11us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 14:47:51.225651 (+ 115us) server_negotiation.cc:1035] Waiting for connection context
0620 14:47:51.225783 (+ 132us) server_negotiation.cc:299] Negotiation successful
0620 14:47:51.225839 (+ 56us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":92}
I20260620 14:47:51.234665 5197 raft_consensus.cc:1261] T 93bee0eb11a44bd399865099b5b51f23 P 907857c6c4bb416299ab3c558ad9bf04 [term 1 FOLLOWER]: Refusing update from remote peer fe615ae6a4174b948f8d28e970eb94a2: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 14:47:51.234700 5458 raft_consensus.cc:1261] T 93bee0eb11a44bd399865099b5b51f23 P 212b7843c32b4cca929902031c380a58 [term 1 FOLLOWER]: Refusing update from remote peer fe615ae6a4174b948f8d28e970eb94a2: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 14:47:51.235803 5528 consensus_queue.cc:1035] T 93bee0eb11a44bd399865099b5b51f23 P fe615ae6a4174b948f8d28e970eb94a2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "212b7843c32b4cca929902031c380a58" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 45123 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 14:47:51.236089 5532 consensus_queue.cc:1035] T 93bee0eb11a44bd399865099b5b51f23 P fe615ae6a4174b948f8d28e970eb94a2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "907857c6c4bb416299ab3c558ad9bf04" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 40157 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 14:47:51.246239 5535 mvcc.cc:205] Tried to move back new op lower bound from 7298936304566747136 to 7298936304411512832. Current Snapshot: MvccSnapshot[applied={T|T < 7298936304566747136}]
I20260620 14:47:51.249609 5537 mvcc.cc:205] Tried to move back new op lower bound from 7298936304566747136 to 7298936304411512832. Current Snapshot: MvccSnapshot[applied={T|T < 7298936304566747136}]
I20260620 14:47:51.250206 5536 mvcc.cc:205] Tried to move back new op lower bound from 7298936304566747136 to 7298936304411512832. Current Snapshot: MvccSnapshot[applied={T|T < 7298936304566747136}]
I20260620 14:47:51.263579 5055 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:40902:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 14:47:51.263921 5055 catalog_manager.cc:2645] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:40902:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 14:47:51.268066 5055 catalog_manager.cc:5782] T 00000000000000000000000000000000 P eab9fc0bea2147b995dd1d8b3db3a5eb: Sending DeleteTablet for 3 replicas of tablet 93bee0eb11a44bd399865099b5b51f23
I20260620 14:47:51.269384 5308 tablet_service.cc:1514] Processing DeleteTablet for tablet 93bee0eb11a44bd399865099b5b51f23 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 14:47:51 UTC) from {username='slave'} at 127.0.0.1:36042
I20260620 14:47:51.269395 5177 tablet_service.cc:1514] Processing DeleteTablet for tablet 93bee0eb11a44bd399865099b5b51f23 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 14:47:51 UTC) from {username='slave'} at 127.0.0.1:57860
I20260620 14:47:51.270134 5438 tablet_service.cc:1514] Processing DeleteTablet for tablet 93bee0eb11a44bd399865099b5b51f23 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 14:47:51 UTC) from {username='slave'} at 127.0.0.1:33850
I20260620 14:47:51.277367 5546 tablet_replica.cc:330] T 93bee0eb11a44bd399865099b5b51f23 P 212b7843c32b4cca929902031c380a58: stopping tablet replica
I20260620 14:47:51.277973 5546 raft_consensus.cc:2229] T 93bee0eb11a44bd399865099b5b51f23 P 212b7843c32b4cca929902031c380a58 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 14:47:51.278640 5546 raft_consensus.cc:2258] T 93bee0eb11a44bd399865099b5b51f23 P 212b7843c32b4cca929902031c380a58 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 14:47:51.279911 5544 tablet_replica.cc:330] T 93bee0eb11a44bd399865099b5b51f23 P 907857c6c4bb416299ab3c558ad9bf04: stopping tablet replica
I20260620 14:47:51.280521 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskUci3DU/build/asan/bin/kudu pid 5115
I20260620 14:47:51.280476 5544 raft_consensus.cc:2229] T 93bee0eb11a44bd399865099b5b51f23 P 907857c6c4bb416299ab3c558ad9bf04 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 14:47:51.281111 5544 raft_consensus.cc:2258] T 93bee0eb11a44bd399865099b5b51f23 P 907857c6c4bb416299ab3c558ad9bf04 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 14:47:51.281098 5546 ts_tablet_manager.cc:1905] T 93bee0eb11a44bd399865099b5b51f23 P 212b7843c32b4cca929902031c380a58: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 14:47:51.283248 5544 ts_tablet_manager.cc:1905] T 93bee0eb11a44bd399865099b5b51f23 P 907857c6c4bb416299ab3c558ad9bf04: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 14:47:51.288429 5546 ts_tablet_manager.cc:1918] T 93bee0eb11a44bd399865099b5b51f23 P 212b7843c32b4cca929902031c380a58: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260620 14:47:51.288595 5546 log.cc:1196] T 93bee0eb11a44bd399865099b5b51f23 P 212b7843c32b4cca929902031c380a58: Deleting WAL directory at /tmp/dist-test-taskUci3DU/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781966829091300-388-0/minicluster-data/ts-2/wal/wals/93bee0eb11a44bd399865099b5b51f23
I20260620 14:47:51.289042 5546 ts_tablet_manager.cc:1939] T 93bee0eb11a44bd399865099b5b51f23 P 212b7843c32b4cca929902031c380a58: Deleting consensus metadata
I20260620 14:47:51.290876 5040 catalog_manager.cc:4841] TS 212b7843c32b4cca929902031c380a58 (127.0.97.3:45123): tablet 93bee0eb11a44bd399865099b5b51f23 (table test-table [id=79204da38ca14f569d18f1e22107c116]) successfully deleted
I20260620 14:47:51.295756 5544 ts_tablet_manager.cc:1918] T 93bee0eb11a44bd399865099b5b51f23 P 907857c6c4bb416299ab3c558ad9bf04: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260620 14:47:51.295979 5544 log.cc:1196] T 93bee0eb11a44bd399865099b5b51f23 P 907857c6c4bb416299ab3c558ad9bf04: Deleting WAL directory at /tmp/dist-test-taskUci3DU/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781966829091300-388-0/minicluster-data/ts-0/wal/wals/93bee0eb11a44bd399865099b5b51f23
I20260620 14:47:51.296525 5544 ts_tablet_manager.cc:1939] T 93bee0eb11a44bd399865099b5b51f23 P 907857c6c4bb416299ab3c558ad9bf04: Deleting consensus metadata
I20260620 14:47:51.298413 5040 catalog_manager.cc:4841] TS 907857c6c4bb416299ab3c558ad9bf04 (127.0.97.1:40157): tablet 93bee0eb11a44bd399865099b5b51f23 (table test-table [id=79204da38ca14f569d18f1e22107c116]) successfully deleted
I20260620 14:47:51.299407 5545 tablet_replica.cc:330] T 93bee0eb11a44bd399865099b5b51f23 P fe615ae6a4174b948f8d28e970eb94a2: stopping tablet replica
I20260620 14:47:51.300841 5545 raft_consensus.cc:2229] T 93bee0eb11a44bd399865099b5b51f23 P fe615ae6a4174b948f8d28e970eb94a2 [term 1 LEADER]: Raft consensus shutting down.
I20260620 14:47:51.301177 5516 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:47:51.282007 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57868 (local address 127.0.97.1:40157)
0620 14:47:51.282197 (+ 190us) server_negotiation.cc:206] Beginning negotiation
0620 14:47:51.282203 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 14:47:51.298639 (+ 16436us) server_negotiation.cc:407] Connection header received
0620 14:47:51.298947 (+ 308us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:47:51.298954 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:47:51.299065 (+ 111us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:47:51.299148 (+ 83us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:47:51.300382 (+ 1234us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 14:47:51.300387 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 14:47:51.300392 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 14:47:51.300411 (+ 19us) server_negotiation.cc:922] calling sasl_server_start()
0620 14:47:51.300448 (+ 37us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 14:47:51.300464 (+ 16us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 14:47:51.300563 (+ 99us) server_negotiation.cc:1035] Waiting for connection context
0620 14:47:51.300801 (+ 238us) server_negotiation.cc:299] Negotiation successful
0620 14:47:51.300869 (+ 68us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":71}
I20260620 14:47:51.301707 5545 raft_consensus.cc:2258] T 93bee0eb11a44bd399865099b5b51f23 P fe615ae6a4174b948f8d28e970eb94a2 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 14:47:51.302233 5237 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 14:47:51.303676 5545 ts_tablet_manager.cc:1905] T 93bee0eb11a44bd399865099b5b51f23 P fe615ae6a4174b948f8d28e970eb94a2: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 14:47:51.311349 5545 ts_tablet_manager.cc:1918] T 93bee0eb11a44bd399865099b5b51f23 P fe615ae6a4174b948f8d28e970eb94a2: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260620 14:47:51.311702 5545 log.cc:1196] T 93bee0eb11a44bd399865099b5b51f23 P fe615ae6a4174b948f8d28e970eb94a2: Deleting WAL directory at /tmp/dist-test-taskUci3DU/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781966829091300-388-0/minicluster-data/ts-1/wal/wals/93bee0eb11a44bd399865099b5b51f23
I20260620 14:47:51.312227 5545 ts_tablet_manager.cc:1939] T 93bee0eb11a44bd399865099b5b51f23 P fe615ae6a4174b948f8d28e970eb94a2: Deleting consensus metadata
I20260620 14:47:51.313917 5041 catalog_manager.cc:4841] TS fe615ae6a4174b948f8d28e970eb94a2 (127.0.97.2:37789): tablet 93bee0eb11a44bd399865099b5b51f23 (table test-table [id=79204da38ca14f569d18f1e22107c116]) successfully deleted
I20260620 14:47:51.402608 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskUci3DU/build/asan/bin/kudu with pid 5115
I20260620 14:47:51.419236 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskUci3DU/build/asan/bin/kudu pid 5246
I20260620 14:47:51.422340 5518 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:47:51.420296 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36054 (local address 127.0.97.2:37789)
0620 14:47:51.420472 (+ 176us) server_negotiation.cc:206] Beginning negotiation
0620 14:47:51.420479 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0620 14:47:51.420497 (+ 18us) server_negotiation.cc:407] Connection header received
0620 14:47:51.420580 (+ 83us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:47:51.420586 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:47:51.420661 (+ 75us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:47:51.420772 (+ 111us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:47:51.421656 (+ 884us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 14:47:51.421662 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 14:47:51.421667 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 14:47:51.421686 (+ 19us) server_negotiation.cc:922] calling sasl_server_start()
0620 14:47:51.421725 (+ 39us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 14:47:51.421741 (+ 16us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 14:47:51.421871 (+ 130us) server_negotiation.cc:1035] Waiting for connection context
0620 14:47:51.421943 (+ 72us) server_negotiation.cc:299] Negotiation successful
0620 14:47:51.421998 (+ 55us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":67}
I20260620 14:47:51.423205 5368 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 14:47:51.531188 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskUci3DU/build/asan/bin/kudu with pid 5246
I20260620 14:47:51.549309 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskUci3DU/build/asan/bin/kudu pid 5376
I20260620 14:47:51.551972 5520 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:47:51.550094 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33862 (local address 127.0.97.3:45123)
0620 14:47:51.550266 (+ 172us) server_negotiation.cc:206] Beginning negotiation
0620 14:47:51.550271 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 14:47:51.550288 (+ 17us) server_negotiation.cc:407] Connection header received
0620 14:47:51.550426 (+ 138us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:47:51.550430 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:47:51.550524 (+ 94us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:47:51.550609 (+ 85us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:47:51.551461 (+ 852us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 14:47:51.551464 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 14:47:51.551467 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 14:47:51.551478 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0620 14:47:51.551512 (+ 34us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 14:47:51.551522 (+ 10us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 14:47:51.551669 (+ 147us) server_negotiation.cc:1035] Waiting for connection context
0620 14:47:51.551714 (+ 45us) server_negotiation.cc:299] Negotiation successful
0620 14:47:51.551757 (+ 43us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":56}
I20260620 14:47:51.552606 5498 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 14:47:51.638661 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskUci3DU/build/asan/bin/kudu with pid 5376
I20260620 14:47:51.654532 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskUci3DU/build/asan/bin/kudu pid 5025
I20260620 14:47:51.655773 5086 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 14:47:51.760979 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskUci3DU/build/asan/bin/kudu with pid 5025
2026-06-20T14:47:51Z chronyd exiting
[ OK ] SecurityITest.TestDisableAuthenticationEncryption (2953 ms)
[ RUN ] SecurityITest.TestJwtMiniCluster
I20260620 14:47:51.802187 388 mini_oidc.cc:152] Starting JWKS server
I20260620 14:47:51.804906 388 webserver.cc:460] Webserver started at https://127.0.0.1:45247/ using document root <none> and password file <none>
I20260620 14:47:51.805253 388 mini_oidc.cc:189] Starting OIDC Discovery server
I20260620 14:47:51.805985 388 webserver.cc:460] Webserver started at http://127.0.0.1:39079/ using document root <none> and password file <none>
2026-06-20T14:47:51Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T14:47:51Z Disabled control of system clock
I20260620 14:47:51.834306 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskUci3DU/build/asan/bin/kudu
/tmp/dist-test-taskUci3DU/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskUci3DU/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1781966829091300-388-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskUci3DU/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1781966829091300-388-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskUci3DU/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1781966829091300-388-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskUci3DU/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1781966829091300-388-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:45179
--webserver_interface=127.0.97.62
--webserver_port=0
--builtin_ntp_servers=127.0.97.20:39881
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.97.62:45179
--enable_jwt_token_auth=true
--jwks_url=https://localhost:45247/jwks.json
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--trusted_certificate_file=/tmp/dist-test-taskUci3DU/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1781966829091300-388-0/testchainca.cert with env {}
W20260620 14:47:52.164296 5557 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 14:47:52.164606 5557 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 14:47:52.164695 5557 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 14:47:52.172030 5557 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 14:47:52.172151 5557 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 14:47:52.172191 5557 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 14:47:52.172247 5557 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 14:47:52.172289 5557 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
W20260620 14:47:52.172391 5557 flags.cc:425] Enabled experimental flag: --enable_jwt_token_auth=true
W20260620 14:47:52.172464 5557 flags.cc:425] Enabled experimental flag: --jwks_url=https://localhost:45247/jwks.json
I20260620 14:47:52.182718 5557 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:39881
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskUci3DU/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1781966829091300-388-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskUci3DU/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1781966829091300-388-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.97.62:45179
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:45179
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--enable_jwt_token_auth=true
--jwks_url=https://localhost:45247/jwks.json
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskUci3DU/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1781966829091300-388-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.97.62
--webserver_port=0
--trusted_certificate_file=/tmp/dist-test-taskUci3DU/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1781966829091300-388-0/testchainca.cert
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskUci3DU/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1781966829091300-388-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 13:41:20 UTC on ccycloud.quasar-vdagoi.root.comops.site
build id 4457462
ASAN enabled
I20260620 14:47:52.184084 5557 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 14:47:52.185351 5557 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 14:47:52.197978 5562 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 14:47:52.198002 5563 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 14:47:52.198019 5565 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 14:47:52.198735 5557 server_base.cc:1034] running on GCE node
I20260620 14:47:52.199705 5557 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 14:47:52.201213 5557 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 14:47:52.202471 5557 hybrid_clock.cc:648] HybridClock initialized: now 1781966872202419 us; error 49 us; skew 500 ppm
I20260620 14:47:52.202848 5557 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260620 14:47:52.205313 5557 webserver.cc:460] Webserver started at http://127.0.97.62:36301/ using document root <none> and password file <none>
I20260620 14:47:52.206044 5557 fs_manager.cc:359] Metadata directory not provided
I20260620 14:47:52.206178 5557 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 14:47:52.206558 5557 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 14:47:52.209592 5557 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskUci3DU/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1781966829091300-388-0/minicluster-data/master-0/data/instance:
uuid: "70bee43576f149558fabffd64526cae0"
format_stamp: "Formatted at 2026-06-20 14:47:52 on dist-test-slave-q9j6"
server_key: "6d35fdc7904973868f8c876e7e1f1b27"
server_key_iv: "99ff3b8702350da9e58e2e1d740f4e1a"
server_key_version: "encryptionkey@0"
I20260620 14:47:52.210435 5557 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskUci3DU/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1781966829091300-388-0/minicluster-data/master-0/wal/instance:
uuid: "70bee43576f149558fabffd64526cae0"
format_stamp: "Formatted at 2026-06-20 14:47:52 on dist-test-slave-q9j6"
server_key: "6d35fdc7904973868f8c876e7e1f1b27"
server_key_iv: "99ff3b8702350da9e58e2e1d740f4e1a"
server_key_version: "encryptionkey@0"
I20260620 14:47:52.215721 5557 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.002s
I20260620 14:47:52.219235 5571 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 14:47:52.220752 5557 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.000s sys 0.003s
I20260620 14:47:52.220954 5557 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskUci3DU/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1781966829091300-388-0/minicluster-data/master-0/data,/tmp/dist-test-taskUci3DU/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1781966829091300-388-0/minicluster-data/master-0/wal
uuid: "70bee43576f149558fabffd64526cae0"
format_stamp: "Formatted at 2026-06-20 14:47:52 on dist-test-slave-q9j6"
server_key: "6d35fdc7904973868f8c876e7e1f1b27"
server_key_iv: "99ff3b8702350da9e58e2e1d740f4e1a"
server_key_version: "encryptionkey@0"
I20260620 14:47:52.221184 5557 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskUci3DU/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1781966829091300-388-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskUci3DU/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1781966829091300-388-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskUci3DU/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1781966829091300-388-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 14:47:52.247023 5557 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 14:47:52.248008 5557 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 14:47:52.248307 5557 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 14:47:52.266721 5557 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.62:45179
I20260620 14:47:52.266741 5622 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.62:45179 every 8 connection(s)
I20260620 14:47:52.268170 5557 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskUci3DU/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1781966829091300-388-0/minicluster-data/master-0/data/info.pb
I20260620 14:47:52.272096 5623 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 14:47:52.274744 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskUci3DU/build/asan/bin/kudu as pid 5557
I20260620 14:47:52.274933 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskUci3DU/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1781966829091300-388-0/minicluster-data/master-0/wal/instance
I20260620 14:47:52.275739 388 external_mini_cluster.cc:1428] Setting key 471fd7edba6359aca5a6ad445435310d
I20260620 14:47:52.284734 5623 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 70bee43576f149558fabffd64526cae0: Bootstrap starting.
I20260620 14:47:52.288197 5623 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 70bee43576f149558fabffd64526cae0: Neither blocks nor log segments found. Creating new log.
I20260620 14:47:52.289425 5623 log.cc:824] T 00000000000000000000000000000000 P 70bee43576f149558fabffd64526cae0: Log is configured to *not* fsync() on all Append() calls
I20260620 14:47:52.290014 5626 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:47:52.277839 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40422 (local address 127.0.97.62:45179)
0620 14:47:52.278880 (+ 1041us) server_negotiation.cc:206] Beginning negotiation
0620 14:47:52.278893 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0620 14:47:52.278936 (+ 43us) server_negotiation.cc:407] Connection header received
0620 14:47:52.280727 (+ 1791us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:47:52.280761 (+ 34us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:47:52.281296 (+ 535us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:47:52.281746 (+ 450us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:47:52.282414 (+ 668us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:47:52.283810 (+ 1396us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:47:52.285259 (+ 1449us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:47:52.285941 (+ 682us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:47:52.286172 (+ 231us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:47:52.287434 (+ 1262us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 14:47:52.287443 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 14:47:52.287458 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 14:47:52.288078 (+ 620us) server_negotiation.cc:922] calling sasl_server_start()
0620 14:47:52.288180 (+ 102us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 14:47:52.288205 (+ 25us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 14:47:52.288350 (+ 145us) server_negotiation.cc:1035] Waiting for connection context
0620 14:47:52.288581 (+ 231us) server_negotiation.cc:299] Negotiation successful
0620 14:47:52.288892 (+ 311us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":573,"thread_start_us":315,"threads_started":1}
I20260620 14:47:52.292675 5623 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 70bee43576f149558fabffd64526cae0: No bootstrap required, opened a new log
I20260620 14:47:52.294091 388 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260620 14:47:52.300424 5623 raft_consensus.cc:348] T 00000000000000000000000000000000 P 70bee43576f149558fabffd64526cae0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "70bee43576f149558fabffd64526cae0" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 45179 } }
I20260620 14:47:52.300851 5623 raft_consensus.cc:374] T 00000000000000000000000000000000 P 70bee43576f149558fabffd64526cae0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 14:47:52.300953 5623 raft_consensus.cc:729] T 00000000000000000000000000000000 P 70bee43576f149558fabffd64526cae0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 70bee43576f149558fabffd64526cae0, State: Initialized, Role: FOLLOWER
I20260620 14:47:52.301636 5623 consensus_queue.cc:260] T 00000000000000000000000000000000 P 70bee43576f149558fabffd64526cae0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "70bee43576f149558fabffd64526cae0" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 45179 } }
I20260620 14:47:52.301880 5623 raft_consensus.cc:388] T 00000000000000000000000000000000 P 70bee43576f149558fabffd64526cae0 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 14:47:52.302007 5623 raft_consensus.cc:482] T 00000000000000000000000000000000 P 70bee43576f149558fabffd64526cae0 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 14:47:52.302177 5623 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 70bee43576f149558fabffd64526cae0 [term 0 FOLLOWER]: Advancing to term 1
I20260620 14:47:52.305557 5623 raft_consensus.cc:504] T 00000000000000000000000000000000 P 70bee43576f149558fabffd64526cae0 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "70bee43576f149558fabffd64526cae0" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 45179 } }
I20260620 14:47:52.306170 5623 leader_election.cc:302] T 00000000000000000000000000000000 P 70bee43576f149558fabffd64526cae0 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 70bee43576f149558fabffd64526cae0; no voters:
I20260620 14:47:52.306761 5623 leader_election.cc:288] T 00000000000000000000000000000000 P 70bee43576f149558fabffd64526cae0 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260620 14:47:52.306946 5628 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 70bee43576f149558fabffd64526cae0 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 14:47:52.307646 5628 raft_consensus.cc:686] T 00000000000000000000000000000000 P 70bee43576f149558fabffd64526cae0 [term 1 LEADER]: Becoming Leader. State: Replica: 70bee43576f149558fabffd64526cae0, State: Running, Role: LEADER
I20260620 14:47:52.308229 5628 consensus_queue.cc:237] T 00000000000000000000000000000000 P 70bee43576f149558fabffd64526cae0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "70bee43576f149558fabffd64526cae0" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 45179 } }
I20260620 14:47:52.309064 5623 sys_catalog.cc:564] T 00000000000000000000000000000000 P 70bee43576f149558fabffd64526cae0 [sys.catalog]: configured and running, proceeding with master startup.
I20260620 14:47:52.311688 5629 sys_catalog.cc:455] T 00000000000000000000000000000000 P 70bee43576f149558fabffd64526cae0 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "70bee43576f149558fabffd64526cae0" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "70bee43576f149558fabffd64526cae0" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 45179 } } }
I20260620 14:47:52.312053 5629 sys_catalog.cc:458] T 00000000000000000000000000000000 P 70bee43576f149558fabffd64526cae0 [sys.catalog]: This master's current role is: LEADER
I20260620 14:47:52.312390 5630 sys_catalog.cc:455] T 00000000000000000000000000000000 P 70bee43576f149558fabffd64526cae0 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 70bee43576f149558fabffd64526cae0. Latest consensus state: current_term: 1 leader_uuid: "70bee43576f149558fabffd64526cae0" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "70bee43576f149558fabffd64526cae0" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 45179 } } }
I20260620 14:47:52.312726 5630 sys_catalog.cc:458] T 00000000000000000000000000000000 P 70bee43576f149558fabffd64526cae0 [sys.catalog]: This master's current role is: LEADER
I20260620 14:47:52.313915 5636 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 14:47:52.318683 5636 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 14:47:52.328377 5636 catalog_manager.cc:1347] Generated new cluster ID: 9ef614a83b2b4d2b8a78cd1b96d409bf
I20260620 14:47:52.328524 5636 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 14:47:52.347170 5636 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 14:47:52.348616 5636 catalog_manager.cc:1504] Loading token signing keys...
I20260620 14:47:52.356752 5636 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 70bee43576f149558fabffd64526cae0: Generated new TSK 0
I20260620 14:47:52.357638 5636 catalog_manager.cc:1514] Initializing in-progress tserver states...
E20260620 14:47:52.397346 5653 webserver.cc:533] Webserver: sslize failed: error:14094415:SSL routines:ssl3_read_bytes:sslv3 alert certificate expired
E20260620 14:47:52.397553 5653 webserver.cc:533] Webserver: error shutting down SSL connection: error:140E0197:SSL routines:SSL_shutdown:shutdown while in init
I20260620 14:47:52.398537 5626 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:47:52.376768 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40436 (local address 127.0.97.62:45179)
0620 14:47:52.377034 (+ 266us) server_negotiation.cc:206] Beginning negotiation
0620 14:47:52.377042 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0620 14:47:52.377431 (+ 389us) server_negotiation.cc:407] Connection header received
0620 14:47:52.377794 (+ 363us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:47:52.377798 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:47:52.377873 (+ 75us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:47:52.377987 (+ 114us) server_negotiation.cc:226] Negotiated authn=JWT
0620 14:47:52.378626 (+ 639us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:47:52.379341 (+ 715us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:47:52.381490 (+ 2149us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:47:52.381985 (+ 495us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:47:52.382103 (+ 118us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:47:52.382778 (+ 675us) server_negotiation.cc:365] Received JWT_EXCHANGE NegotiatePB request
0620 14:47:52.397982 (+ 15204us) server_negotiation.cc:394] Sending RPC error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:45247/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
0620 14:47:52.398298 (+ 316us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:40436: Failed to load JWKS: Error downloading JWKS from 'https://localhost:45247/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
Metrics: {"server-negotiator.queue_time_us":81}
W20260620 14:47:52.398684 5626 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:40436: Failed to load JWKS: Error downloading JWKS from 'https://localhost:45247/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
W20260620 14:47:52.400710 5652 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 14:47:52.376653 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:45179 (local address 127.0.0.1:40436)
0620 14:47:52.377312 (+ 659us) negotiation.cc:107] Waiting for socket to connect
0620 14:47:52.377333 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0620 14:47:52.377543 (+ 210us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 14:47:52.378051 (+ 508us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 14:47:52.378057 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 14:47:52.378070 (+ 13us) client_negotiation.cc:189] Negotiated authn=JWT
0620 14:47:52.378458 (+ 388us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:47:52.378469 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:47:52.379484 (+ 1015us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:47:52.379492 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:47:52.381313 (+ 1821us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:47:52.381329 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:47:52.382121 (+ 792us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:47:52.382196 (+ 75us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:47:52.382391 (+ 195us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:47:52.382494 (+ 103us) client_negotiation.cc:252] Sending JWT_EXCHANGE NegotiatePB request
0620 14:47:52.398314 (+ 15820us) client_negotiation.cc:283] Received error response from server: Runtime error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:45247/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
0620 14:47:52.398715 (+ 401us) negotiation.cc:327] Negotiation complete: Runtime error: Client connection negotiation failed: client connection to 127.0.97.62:45179: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:45247/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
Metrics: {"client-negotiator.queue_time_us":461,"thread_start_us":320,"threads_started":1}
/data0/somelongdirectorytoavoidrpathissues/src/kudu/src/kudu/integration-tests/security-itest.cc:597: Failure
Failed
Bad status: Runtime error: Could not connect to the cluster: Client connection negotiation failed: client connection to 127.0.97.62:45179: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:45247/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
Google Test trace:
/data0/somelongdirectorytoavoidrpathissues/src/kudu/src/kudu/integration-tests/security-itest.cc:593: Valid JWT
I20260620 14:47:52.402693 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskUci3DU/build/asan/bin/kudu pid 5557
I20260620 14:47:52.404008 5618 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 14:47:52.498211 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskUci3DU/build/asan/bin/kudu with pid 5557
2026-06-20T14:47:52Z chronyd exiting
I20260620 14:47:52.548996 388 test_util.cc:175] -----------------------------------------------
I20260620 14:47:52.549151 388 test_util.cc:176] Had fatal failures, leaving test files at /tmp/dist-test-taskUci3DU/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1781966829091300-388-0
[ FAILED ] SecurityITest.TestJwtMiniCluster (751 ms)
[ RUN ] SecurityITest.TestJwtMiniClusterWithInvalidCert
I20260620 14:47:52.552932 388 mini_oidc.cc:152] Starting JWKS server
I20260620 14:47:52.554739 388 webserver.cc:460] Webserver started at https://127.0.0.1:42615/ using document root <none> and password file <none>
I20260620 14:47:52.554993 388 mini_oidc.cc:189] Starting OIDC Discovery server
I20260620 14:47:52.555632 388 webserver.cc:460] Webserver started at http://127.0.0.1:36633/ using document root <none> and password file <none>
2026-06-20T14:47:52Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T14:47:52Z Disabled control of system clock
I20260620 14:47:52.582656 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskUci3DU/build/asan/bin/kudu
/tmp/dist-test-taskUci3DU/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskUci3DU/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1781966829091300-388-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskUci3DU/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1781966829091300-388-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskUci3DU/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1781966829091300-388-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskUci3DU/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1781966829091300-388-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:35841
--webserver_interface=127.0.97.62
--webserver_port=0
--builtin_ntp_servers=127.0.97.20:35385
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.97.62:35841
--enable_jwt_token_auth=true
--jwks_url=https://localhost:42615/jwks.json
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--trusted_certificate_file=/tmp/dist-test-taskUci3DU/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1781966829091300-388-0/testchainca.cert with env {}
W20260620 14:47:52.909869 5660 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 14:47:52.910169 5660 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 14:47:52.910255 5660 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 14:47:52.917779 5660 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 14:47:52.917906 5660 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 14:47:52.917943 5660 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 14:47:52.917992 5660 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 14:47:52.918036 5660 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
W20260620 14:47:52.918092 5660 flags.cc:425] Enabled experimental flag: --enable_jwt_token_auth=true
W20260620 14:47:52.918129 5660 flags.cc:425] Enabled experimental flag: --jwks_url=https://localhost:42615/jwks.json
I20260620 14:47:52.928180 5660 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:35385
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskUci3DU/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1781966829091300-388-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskUci3DU/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1781966829091300-388-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.97.62:35841
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:35841
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--enable_jwt_token_auth=true
--jwks_url=https://localhost:42615/jwks.json
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskUci3DU/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1781966829091300-388-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.97.62
--webserver_port=0
--trusted_certificate_file=/tmp/dist-test-taskUci3DU/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1781966829091300-388-0/testchainca.cert
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskUci3DU/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1781966829091300-388-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 13:41:20 UTC on ccycloud.quasar-vdagoi.root.comops.site
build id 4457462
ASAN enabled
I20260620 14:47:52.929454 5660 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 14:47:52.930716 5660 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 14:47:52.942528 5665 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 14:47:52.942600 5666 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 14:47:52.942656 5668 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 14:47:52.943555 5660 server_base.cc:1034] running on GCE node
I20260620 14:47:52.944412 5660 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 14:47:52.945879 5660 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 14:47:52.947118 5660 hybrid_clock.cc:648] HybridClock initialized: now 1781966872947083 us; error 60 us; skew 500 ppm
I20260620 14:47:52.947482 5660 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260620 14:47:52.949703 5660 webserver.cc:460] Webserver started at http://127.0.97.62:44937/ using document root <none> and password file <none>
I20260620 14:47:52.950479 5660 fs_manager.cc:359] Metadata directory not provided
I20260620 14:47:52.950613 5660 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 14:47:52.950938 5660 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 14:47:52.953838 5660 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskUci3DU/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1781966829091300-388-0/minicluster-data/master-0/data/instance:
uuid: "b374c227f5b943fc8c18e67c2b55ee85"
format_stamp: "Formatted at 2026-06-20 14:47:52 on dist-test-slave-q9j6"
server_key: "dde3f12fe03280aa217846ec8d063f4a"
server_key_iv: "8cd2dc8e42bca5071f796bd056b6e32d"
server_key_version: "encryptionkey@0"
I20260620 14:47:52.954677 5660 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskUci3DU/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1781966829091300-388-0/minicluster-data/master-0/wal/instance:
uuid: "b374c227f5b943fc8c18e67c2b55ee85"
format_stamp: "Formatted at 2026-06-20 14:47:52 on dist-test-slave-q9j6"
server_key: "dde3f12fe03280aa217846ec8d063f4a"
server_key_iv: "8cd2dc8e42bca5071f796bd056b6e32d"
server_key_version: "encryptionkey@0"
I20260620 14:47:52.959877 5660 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.008s sys 0.000s
I20260620 14:47:52.963476 5674 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 14:47:52.964859 5660 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260620 14:47:52.965062 5660 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskUci3DU/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1781966829091300-388-0/minicluster-data/master-0/data,/tmp/dist-test-taskUci3DU/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1781966829091300-388-0/minicluster-data/master-0/wal
uuid: "b374c227f5b943fc8c18e67c2b55ee85"
format_stamp: "Formatted at 2026-06-20 14:47:52 on dist-test-slave-q9j6"
server_key: "dde3f12fe03280aa217846ec8d063f4a"
server_key_iv: "8cd2dc8e42bca5071f796bd056b6e32d"
server_key_version: "encryptionkey@0"
I20260620 14:47:52.965286 5660 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskUci3DU/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1781966829091300-388-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskUci3DU/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1781966829091300-388-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskUci3DU/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1781966829091300-388-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 14:47:52.977149 5660 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 14:47:52.978000 5660 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 14:47:52.978291 5660 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 14:47:52.996490 5660 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.62:35841
I20260620 14:47:52.996513 5725 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.62:35841 every 8 connection(s)
I20260620 14:47:52.997850 5660 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskUci3DU/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1781966829091300-388-0/minicluster-data/master-0/data/info.pb
I20260620 14:47:53.001426 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskUci3DU/build/asan/bin/kudu as pid 5660
I20260620 14:47:53.001653 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskUci3DU/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1781966829091300-388-0/minicluster-data/master-0/wal/instance
I20260620 14:47:53.001735 5726 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 14:47:53.002449 388 external_mini_cluster.cc:1428] Setting key f7c9db05ca18aa800b526cc6a72c1560
I20260620 14:47:53.018090 5726 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b374c227f5b943fc8c18e67c2b55ee85: Bootstrap starting.
I20260620 14:47:53.018214 5729 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:47:53.004184 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34076 (local address 127.0.97.62:35841)
0620 14:47:53.005250 (+ 1066us) server_negotiation.cc:206] Beginning negotiation
0620 14:47:53.005266 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0620 14:47:53.005323 (+ 57us) server_negotiation.cc:407] Connection header received
0620 14:47:53.007413 (+ 2090us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:47:53.007454 (+ 41us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:47:53.008191 (+ 737us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:47:53.008706 (+ 515us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:47:53.009317 (+ 611us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:47:53.011338 (+ 2021us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:47:53.012692 (+ 1354us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:47:53.013799 (+ 1107us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:47:53.013991 (+ 192us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:47:53.015141 (+ 1150us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 14:47:53.015152 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 14:47:53.015172 (+ 20us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 14:47:53.016003 (+ 831us) server_negotiation.cc:922] calling sasl_server_start()
0620 14:47:53.016103 (+ 100us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 14:47:53.016129 (+ 26us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 14:47:53.016244 (+ 115us) server_negotiation.cc:1035] Waiting for connection context
0620 14:47:53.016495 (+ 251us) server_negotiation.cc:299] Negotiation successful
0620 14:47:53.016849 (+ 354us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":536,"thread_start_us":337,"threads_started":1}
I20260620 14:47:53.022061 5726 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P b374c227f5b943fc8c18e67c2b55ee85: Neither blocks nor log segments found. Creating new log.
I20260620 14:47:53.022528 388 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260620 14:47:53.023243 5726 log.cc:824] T 00000000000000000000000000000000 P b374c227f5b943fc8c18e67c2b55ee85: Log is configured to *not* fsync() on all Append() calls
I20260620 14:47:53.026968 5726 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b374c227f5b943fc8c18e67c2b55ee85: No bootstrap required, opened a new log
I20260620 14:47:53.035555 5726 raft_consensus.cc:348] T 00000000000000000000000000000000 P b374c227f5b943fc8c18e67c2b55ee85 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b374c227f5b943fc8c18e67c2b55ee85" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 35841 } }
I20260620 14:47:53.036103 5726 raft_consensus.cc:374] T 00000000000000000000000000000000 P b374c227f5b943fc8c18e67c2b55ee85 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 14:47:53.036247 5726 raft_consensus.cc:729] T 00000000000000000000000000000000 P b374c227f5b943fc8c18e67c2b55ee85 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b374c227f5b943fc8c18e67c2b55ee85, State: Initialized, Role: FOLLOWER
I20260620 14:47:53.036798 5726 consensus_queue.cc:260] T 00000000000000000000000000000000 P b374c227f5b943fc8c18e67c2b55ee85 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b374c227f5b943fc8c18e67c2b55ee85" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 35841 } }
I20260620 14:47:53.037058 5726 raft_consensus.cc:388] T 00000000000000000000000000000000 P b374c227f5b943fc8c18e67c2b55ee85 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 14:47:53.037179 5726 raft_consensus.cc:482] T 00000000000000000000000000000000 P b374c227f5b943fc8c18e67c2b55ee85 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 14:47:53.037312 5726 raft_consensus.cc:3037] T 00000000000000000000000000000000 P b374c227f5b943fc8c18e67c2b55ee85 [term 0 FOLLOWER]: Advancing to term 1
I20260620 14:47:53.040053 5726 raft_consensus.cc:504] T 00000000000000000000000000000000 P b374c227f5b943fc8c18e67c2b55ee85 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b374c227f5b943fc8c18e67c2b55ee85" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 35841 } }
I20260620 14:47:53.040588 5726 leader_election.cc:302] T 00000000000000000000000000000000 P b374c227f5b943fc8c18e67c2b55ee85 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: b374c227f5b943fc8c18e67c2b55ee85; no voters:
I20260620 14:47:53.041143 5726 leader_election.cc:288] T 00000000000000000000000000000000 P b374c227f5b943fc8c18e67c2b55ee85 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260620 14:47:53.041286 5731 raft_consensus.cc:2781] T 00000000000000000000000000000000 P b374c227f5b943fc8c18e67c2b55ee85 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 14:47:53.041991 5731 raft_consensus.cc:686] T 00000000000000000000000000000000 P b374c227f5b943fc8c18e67c2b55ee85 [term 1 LEADER]: Becoming Leader. State: Replica: b374c227f5b943fc8c18e67c2b55ee85, State: Running, Role: LEADER
I20260620 14:47:53.042614 5731 consensus_queue.cc:237] T 00000000000000000000000000000000 P b374c227f5b943fc8c18e67c2b55ee85 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b374c227f5b943fc8c18e67c2b55ee85" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 35841 } }
I20260620 14:47:53.043365 5726 sys_catalog.cc:564] T 00000000000000000000000000000000 P b374c227f5b943fc8c18e67c2b55ee85 [sys.catalog]: configured and running, proceeding with master startup.
I20260620 14:47:53.045275 5732 sys_catalog.cc:455] T 00000000000000000000000000000000 P b374c227f5b943fc8c18e67c2b55ee85 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "b374c227f5b943fc8c18e67c2b55ee85" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b374c227f5b943fc8c18e67c2b55ee85" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 35841 } } }
I20260620 14:47:53.045679 5732 sys_catalog.cc:458] T 00000000000000000000000000000000 P b374c227f5b943fc8c18e67c2b55ee85 [sys.catalog]: This master's current role is: LEADER
I20260620 14:47:53.050370 5733 sys_catalog.cc:455] T 00000000000000000000000000000000 P b374c227f5b943fc8c18e67c2b55ee85 [sys.catalog]: SysCatalogTable state changed. Reason: New leader b374c227f5b943fc8c18e67c2b55ee85. Latest consensus state: current_term: 1 leader_uuid: "b374c227f5b943fc8c18e67c2b55ee85" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b374c227f5b943fc8c18e67c2b55ee85" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 35841 } } }
I20260620 14:47:53.050772 5733 sys_catalog.cc:458] T 00000000000000000000000000000000 P b374c227f5b943fc8c18e67c2b55ee85 [sys.catalog]: This master's current role is: LEADER
I20260620 14:47:53.051705 5736 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 14:47:53.058630 5736 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 14:47:53.070379 5736 catalog_manager.cc:1347] Generated new cluster ID: 48e6fd056ddc4e6fa94305532103f7b9
I20260620 14:47:53.070516 5736 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 14:47:53.089129 5736 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 14:47:53.091038 5736 catalog_manager.cc:1504] Loading token signing keys...
I20260620 14:47:53.104584 5736 catalog_manager.cc:5868] T 00000000000000000000000000000000 P b374c227f5b943fc8c18e67c2b55ee85: Generated new TSK 0
I20260620 14:47:53.105674 5736 catalog_manager.cc:1514] Initializing in-progress tserver states...
E20260620 14:47:53.151000 5756 webserver.cc:533] Webserver: sslize failed: error:14094418:SSL routines:ssl3_read_bytes:tlsv1 alert unknown ca
E20260620 14:47:53.151149 5756 webserver.cc:533] Webserver: error shutting down SSL connection: error:140E0197:SSL routines:SSL_shutdown:shutdown while in init
I20260620 14:47:53.152534 5729 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:47:53.131636 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34084 (local address 127.0.97.62:35841)
0620 14:47:53.131834 (+ 198us) server_negotiation.cc:206] Beginning negotiation
0620 14:47:53.131842 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0620 14:47:53.132288 (+ 446us) server_negotiation.cc:407] Connection header received
0620 14:47:53.132541 (+ 253us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:47:53.132547 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:47:53.132639 (+ 92us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:47:53.132812 (+ 173us) server_negotiation.cc:226] Negotiated authn=JWT
0620 14:47:53.133254 (+ 442us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:47:53.134426 (+ 1172us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:47:53.135885 (+ 1459us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:47:53.136649 (+ 764us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:47:53.136791 (+ 142us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:47:53.137453 (+ 662us) server_negotiation.cc:365] Received JWT_EXCHANGE NegotiatePB request
0620 14:47:53.151778 (+ 14325us) server_negotiation.cc:394] Sending RPC error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:42615/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: unable to get local issuer certificate
0620 14:47:53.152206 (+ 428us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:34084: Failed to load JWKS: Error downloading JWKS from 'https://localhost:42615/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: unable to get local issuer certificate
Metrics: {"server-negotiator.queue_time_us":62}
W20260620 14:47:53.152549 5755 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 14:47:53.131435 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:35841 (local address 127.0.0.1:34084)
0620 14:47:53.132182 (+ 747us) negotiation.cc:107] Waiting for socket to connect
0620 14:47:53.132202 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0620 14:47:53.132324 (+ 122us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 14:47:53.132829 (+ 505us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 14:47:53.132832 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 14:47:53.132841 (+ 9us) client_negotiation.cc:189] Negotiated authn=JWT
0620 14:47:53.133091 (+ 250us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:47:53.133098 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:47:53.134604 (+ 1506us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:47:53.134608 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:47:53.135747 (+ 1139us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:47:53.135759 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:47:53.136821 (+ 1062us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:47:53.136878 (+ 57us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:47:53.137087 (+ 209us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:47:53.137137 (+ 50us) client_negotiation.cc:252] Sending JWT_EXCHANGE NegotiatePB request
0620 14:47:53.152046 (+ 14909us) client_negotiation.cc:283] Received error response from server: Runtime error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:42615/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: unable to get local issuer certificate
0620 14:47:53.152254 (+ 208us) negotiation.cc:327] Negotiation complete: Runtime error: Client connection negotiation failed: client connection to 127.0.97.62:35841: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:42615/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: unable to get local issuer certificate
Metrics: {"client-negotiator.queue_time_us":612,"thread_start_us":458,"threads_started":1}
W20260620 14:47:53.152669 5729 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:34084: Failed to load JWKS: Error downloading JWKS from 'https://localhost:42615/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: unable to get local issuer certificate
I20260620 14:47:53.154119 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskUci3DU/build/asan/bin/kudu pid 5660
I20260620 14:47:53.155298 5721 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 14:47:53.250218 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskUci3DU/build/asan/bin/kudu with pid 5660
2026-06-20T14:47:53Z chronyd exiting
[ OK ] SecurityITest.TestJwtMiniClusterWithInvalidCert (755 ms)
[ RUN ] SecurityITest.TestJwtMiniClusterWithUntrustedCert
I20260620 14:47:53.308205 388 mini_oidc.cc:152] Starting JWKS server
I20260620 14:47:53.310127 388 webserver.cc:460] Webserver started at https://127.0.0.1:44839/ using document root <none> and password file <none>
I20260620 14:47:53.310361 388 mini_oidc.cc:189] Starting OIDC Discovery server
I20260620 14:47:53.310992 388 webserver.cc:460] Webserver started at http://127.0.0.1:44845/ using document root <none> and password file <none>
2026-06-20T14:47:53Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T14:47:53Z Disabled control of system clock
I20260620 14:47:53.338817 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskUci3DU/build/asan/bin/kudu
/tmp/dist-test-taskUci3DU/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskUci3DU/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1781966829091300-388-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskUci3DU/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1781966829091300-388-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskUci3DU/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1781966829091300-388-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskUci3DU/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1781966829091300-388-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:41567
--webserver_interface=127.0.97.62
--webserver_port=0
--builtin_ntp_servers=127.0.97.20:42975
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.97.62:41567
--enable_jwt_token_auth=true
--jwks_url=https://localhost:44839/jwks.json
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {}
W20260620 14:47:53.666618 5763 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 14:47:53.666931 5763 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 14:47:53.667018 5763 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 14:47:53.674319 5763 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 14:47:53.674445 5763 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 14:47:53.674491 5763 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 14:47:53.674541 5763 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 14:47:53.674587 5763 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
W20260620 14:47:53.674642 5763 flags.cc:425] Enabled experimental flag: --enable_jwt_token_auth=true
W20260620 14:47:53.674682 5763 flags.cc:425] Enabled experimental flag: --jwks_url=https://localhost:44839/jwks.json
I20260620 14:47:53.683910 5763 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:42975
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskUci3DU/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1781966829091300-388-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskUci3DU/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1781966829091300-388-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.97.62:41567
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:41567
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--enable_jwt_token_auth=true
--jwks_url=https://localhost:44839/jwks.json
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskUci3DU/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1781966829091300-388-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.97.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskUci3DU/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1781966829091300-388-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 13:41:20 UTC on ccycloud.quasar-vdagoi.root.comops.site
build id 4457462
ASAN enabled
I20260620 14:47:53.685182 5763 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 14:47:53.686911 5763 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 14:47:53.698873 5768 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 14:47:53.698915 5769 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 14:47:53.698952 5771 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 14:47:53.699919 5763 server_base.cc:1034] running on GCE node
I20260620 14:47:53.700745 5763 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 14:47:53.702294 5763 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 14:47:53.703557 5763 hybrid_clock.cc:648] HybridClock initialized: now 1781966873703489 us; error 61 us; skew 500 ppm
I20260620 14:47:53.703958 5763 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260620 14:47:53.706532 5763 webserver.cc:460] Webserver started at http://127.0.97.62:42361/ using document root <none> and password file <none>
I20260620 14:47:53.707365 5763 fs_manager.cc:359] Metadata directory not provided
I20260620 14:47:53.707499 5763 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 14:47:53.707885 5763 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 14:47:53.710978 5763 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskUci3DU/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1781966829091300-388-0/minicluster-data/master-0/data/instance:
uuid: "73d3128b04c141d9b0212efb709d7058"
format_stamp: "Formatted at 2026-06-20 14:47:53 on dist-test-slave-q9j6"
server_key: "c5decf6997fab606b4adb5962918de5d"
server_key_iv: "1a1925e1dab42c07780f068c0f0efdc2"
server_key_version: "encryptionkey@0"
I20260620 14:47:53.711943 5763 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskUci3DU/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1781966829091300-388-0/minicluster-data/master-0/wal/instance:
uuid: "73d3128b04c141d9b0212efb709d7058"
format_stamp: "Formatted at 2026-06-20 14:47:53 on dist-test-slave-q9j6"
server_key: "c5decf6997fab606b4adb5962918de5d"
server_key_iv: "1a1925e1dab42c07780f068c0f0efdc2"
server_key_version: "encryptionkey@0"
I20260620 14:47:53.717375 5763 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.004s sys 0.000s
I20260620 14:47:53.721072 5777 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 14:47:53.722611 5763 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260620 14:47:53.722827 5763 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskUci3DU/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1781966829091300-388-0/minicluster-data/master-0/data,/tmp/dist-test-taskUci3DU/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1781966829091300-388-0/minicluster-data/master-0/wal
uuid: "73d3128b04c141d9b0212efb709d7058"
format_stamp: "Formatted at 2026-06-20 14:47:53 on dist-test-slave-q9j6"
server_key: "c5decf6997fab606b4adb5962918de5d"
server_key_iv: "1a1925e1dab42c07780f068c0f0efdc2"
server_key_version: "encryptionkey@0"
I20260620 14:47:53.723071 5763 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskUci3DU/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1781966829091300-388-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskUci3DU/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1781966829091300-388-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskUci3DU/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1781966829091300-388-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 14:47:53.757618 5763 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 14:47:53.758574 5763 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 14:47:53.758867 5763 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 14:47:53.777266 5763 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.62:41567
I20260620 14:47:53.777279 5828 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.62:41567 every 8 connection(s)
I20260620 14:47:53.778796 5763 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskUci3DU/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1781966829091300-388-0/minicluster-data/master-0/data/info.pb
I20260620 14:47:53.782591 5829 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 14:47:53.789027 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskUci3DU/build/asan/bin/kudu as pid 5763
I20260620 14:47:53.789242 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskUci3DU/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1781966829091300-388-0/minicluster-data/master-0/wal/instance
I20260620 14:47:53.789845 388 external_mini_cluster.cc:1428] Setting key eff4e543bdd09c2c9e879fbc0332f477
I20260620 14:47:53.793447 5829 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 73d3128b04c141d9b0212efb709d7058: Bootstrap starting.
I20260620 14:47:53.796866 5829 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 73d3128b04c141d9b0212efb709d7058: Neither blocks nor log segments found. Creating new log.
I20260620 14:47:53.798061 5829 log.cc:824] T 00000000000000000000000000000000 P 73d3128b04c141d9b0212efb709d7058: Log is configured to *not* fsync() on all Append() calls
I20260620 14:47:53.801412 5829 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 73d3128b04c141d9b0212efb709d7058: No bootstrap required, opened a new log
I20260620 14:47:53.803956 5832 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:47:53.791917 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57136 (local address 127.0.97.62:41567)
0620 14:47:53.792902 (+ 985us) server_negotiation.cc:206] Beginning negotiation
0620 14:47:53.792913 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0620 14:47:53.792957 (+ 44us) server_negotiation.cc:407] Connection header received
0620 14:47:53.794687 (+ 1730us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:47:53.794712 (+ 25us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:47:53.795206 (+ 494us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:47:53.795537 (+ 331us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:47:53.796213 (+ 676us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:47:53.797567 (+ 1354us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:47:53.799296 (+ 1729us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:47:53.800097 (+ 801us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:47:53.800235 (+ 138us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:47:53.801423 (+ 1188us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 14:47:53.801432 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 14:47:53.801447 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 14:47:53.802055 (+ 608us) server_negotiation.cc:922] calling sasl_server_start()
0620 14:47:53.802136 (+ 81us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 14:47:53.802161 (+ 25us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 14:47:53.802255 (+ 94us) server_negotiation.cc:1035] Waiting for connection context
0620 14:47:53.802495 (+ 240us) server_negotiation.cc:299] Negotiation successful
0620 14:47:53.802867 (+ 372us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":548,"thread_start_us":344,"threads_started":1}
I20260620 14:47:53.807933 388 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260620 14:47:53.809623 5829 raft_consensus.cc:348] T 00000000000000000000000000000000 P 73d3128b04c141d9b0212efb709d7058 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "73d3128b04c141d9b0212efb709d7058" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 41567 } }
I20260620 14:47:53.810070 5829 raft_consensus.cc:374] T 00000000000000000000000000000000 P 73d3128b04c141d9b0212efb709d7058 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 14:47:53.810177 5829 raft_consensus.cc:729] T 00000000000000000000000000000000 P 73d3128b04c141d9b0212efb709d7058 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 73d3128b04c141d9b0212efb709d7058, State: Initialized, Role: FOLLOWER
I20260620 14:47:53.810797 5829 consensus_queue.cc:260] T 00000000000000000000000000000000 P 73d3128b04c141d9b0212efb709d7058 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "73d3128b04c141d9b0212efb709d7058" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 41567 } }
I20260620 14:47:53.811048 5829 raft_consensus.cc:388] T 00000000000000000000000000000000 P 73d3128b04c141d9b0212efb709d7058 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 14:47:53.811183 5829 raft_consensus.cc:482] T 00000000000000000000000000000000 P 73d3128b04c141d9b0212efb709d7058 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 14:47:53.811359 5829 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 73d3128b04c141d9b0212efb709d7058 [term 0 FOLLOWER]: Advancing to term 1
I20260620 14:47:53.813550 5829 raft_consensus.cc:504] T 00000000000000000000000000000000 P 73d3128b04c141d9b0212efb709d7058 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "73d3128b04c141d9b0212efb709d7058" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 41567 } }
I20260620 14:47:53.814002 5829 leader_election.cc:302] T 00000000000000000000000000000000 P 73d3128b04c141d9b0212efb709d7058 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 73d3128b04c141d9b0212efb709d7058; no voters:
I20260620 14:47:53.814572 5829 leader_election.cc:288] T 00000000000000000000000000000000 P 73d3128b04c141d9b0212efb709d7058 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260620 14:47:53.814704 5834 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 73d3128b04c141d9b0212efb709d7058 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 14:47:53.815515 5834 raft_consensus.cc:686] T 00000000000000000000000000000000 P 73d3128b04c141d9b0212efb709d7058 [term 1 LEADER]: Becoming Leader. State: Replica: 73d3128b04c141d9b0212efb709d7058, State: Running, Role: LEADER
I20260620 14:47:53.816159 5834 consensus_queue.cc:237] T 00000000000000000000000000000000 P 73d3128b04c141d9b0212efb709d7058 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "73d3128b04c141d9b0212efb709d7058" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 41567 } }
I20260620 14:47:53.817011 5829 sys_catalog.cc:564] T 00000000000000000000000000000000 P 73d3128b04c141d9b0212efb709d7058 [sys.catalog]: configured and running, proceeding with master startup.
I20260620 14:47:53.819331 5835 sys_catalog.cc:455] T 00000000000000000000000000000000 P 73d3128b04c141d9b0212efb709d7058 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "73d3128b04c141d9b0212efb709d7058" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "73d3128b04c141d9b0212efb709d7058" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 41567 } } }
I20260620 14:47:53.819769 5835 sys_catalog.cc:458] T 00000000000000000000000000000000 P 73d3128b04c141d9b0212efb709d7058 [sys.catalog]: This master's current role is: LEADER
I20260620 14:47:53.820426 5836 sys_catalog.cc:455] T 00000000000000000000000000000000 P 73d3128b04c141d9b0212efb709d7058 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 73d3128b04c141d9b0212efb709d7058. Latest consensus state: current_term: 1 leader_uuid: "73d3128b04c141d9b0212efb709d7058" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "73d3128b04c141d9b0212efb709d7058" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 41567 } } }
I20260620 14:47:53.820751 5836 sys_catalog.cc:458] T 00000000000000000000000000000000 P 73d3128b04c141d9b0212efb709d7058 [sys.catalog]: This master's current role is: LEADER
I20260620 14:47:53.821672 5840 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 14:47:53.827385 5840 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 14:47:53.835984 5840 catalog_manager.cc:1347] Generated new cluster ID: 9d95a7b4a7d84cb79e5bc37fe7c7295b
I20260620 14:47:53.836124 5840 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 14:47:53.867702 5840 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 14:47:53.869011 5840 catalog_manager.cc:1504] Loading token signing keys...
I20260620 14:47:53.884660 5840 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 73d3128b04c141d9b0212efb709d7058: Generated new TSK 0
I20260620 14:47:53.885854 5840 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260620 14:47:53.919305 5832 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:47:53.911128 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57138 (local address 127.0.97.62:41567)
0620 14:47:53.911336 (+ 208us) server_negotiation.cc:206] Beginning negotiation
0620 14:47:53.911345 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0620 14:47:53.911879 (+ 534us) server_negotiation.cc:407] Connection header received
0620 14:47:53.912170 (+ 291us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:47:53.912174 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:47:53.912246 (+ 72us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:47:53.912425 (+ 179us) server_negotiation.cc:226] Negotiated authn=JWT
0620 14:47:53.912965 (+ 540us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:47:53.913705 (+ 740us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:47:53.915090 (+ 1385us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:47:53.915534 (+ 444us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:47:53.915724 (+ 190us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:47:53.916224 (+ 500us) server_negotiation.cc:365] Received JWT_EXCHANGE NegotiatePB request
0620 14:47:53.918666 (+ 2442us) server_negotiation.cc:394] Sending RPC error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:44839/jwks.json': curl error: Problem with the SSL CA cert (path? access rights?): error setting certificate verify locations:
CAfile: /etc/pki/tls/certs/ca-bundle.crt
CApath: none
0620 14:47:53.918984 (+ 318us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:57138: Failed to load JWKS: Error downloading JWKS from 'https://localhost:44839/jwks.json': curl error: Problem with the SSL CA cert (path? access rights?): error setting certificate verify locations:
CAfile: /etc/pki/tls/certs/ca-bundle.crt
CApath: none
Metrics: {"server-negotiator.queue_time_us":68}
W20260620 14:47:53.919425 5832 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:57138: Failed to load JWKS: Error downloading JWKS from 'https://localhost:44839/jwks.json': curl error: Problem with the SSL CA cert (path? access rights?): error setting certificate verify locations:
CAfile: /etc/pki/tls/certs/ca-bundle.crt
CApath: none
W20260620 14:47:53.921641 5858 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 14:47:53.911103 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:41567 (local address 127.0.0.1:57138)
0620 14:47:53.911769 (+ 666us) negotiation.cc:107] Waiting for socket to connect
0620 14:47:53.911785 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0620 14:47:53.911981 (+ 196us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 14:47:53.912436 (+ 455us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 14:47:53.912447 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 14:47:53.912460 (+ 13us) client_negotiation.cc:189] Negotiated authn=JWT
0620 14:47:53.912807 (+ 347us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:47:53.912817 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:47:53.913852 (+ 1035us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:47:53.913859 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:47:53.914943 (+ 1084us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:47:53.914953 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:47:53.915708 (+ 755us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:47:53.915784 (+ 76us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:47:53.915927 (+ 143us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:47:53.915960 (+ 33us) client_negotiation.cc:252] Sending JWT_EXCHANGE NegotiatePB request
0620 14:47:53.918888 (+ 2928us) client_negotiation.cc:283] Received error response from server: Runtime error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:44839/jwks.json': curl error: Problem with the SSL CA cert (path? access rights?): error setting certificate verify locations:
CAfile: /etc/pki/tls/certs/ca-bundle.crt
CApath: none
0620 14:47:53.921190 (+ 2302us) negotiation.cc:327] Negotiation complete: Runtime error: Client connection negotiation failed: client connection to 127.0.97.62:41567: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:44839/jwks.json': curl error: Problem with the SSL CA cert (path? access rights?): error setting certificate verify locations:
CAfile: /etc/pki/tls/certs/ca-bundle.crt
CApath: none
Metrics: {"client-negotiator.queue_time_us":519,"mutex_wait_us":1972,"thread_start_us":333,"threads_started":1}
I20260620 14:47:53.923306 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskUci3DU/build/asan/bin/kudu pid 5763
I20260620 14:47:53.924623 5824 generic_service.cc:196] Checking for leaks (request via RPC)
E20260620 14:47:53.931741 5859 webserver.cc:533] Webserver: sslize failed:
E20260620 14:47:53.931871 5859 webserver.cc:533] Webserver: error shutting down SSL connection: error:140E0197:SSL routines:SSL_shutdown:shutdown while in init
I20260620 14:47:54.017601 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskUci3DU/build/asan/bin/kudu with pid 5763
2026-06-20T14:47:54Z chronyd exiting
[ OK ] SecurityITest.TestJwtMiniClusterWithUntrustedCert (766 ms)
[ RUN ] SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS
2026-06-20T14:47:54Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T14:47:54Z Disabled control of system clock
I20260620 14:47:54.103734 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskUci3DU/build/asan/bin/kudu
/tmp/dist-test-taskUci3DU/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskUci3DU/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1781966829091300-388-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskUci3DU/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1781966829091300-388-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskUci3DU/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1781966829091300-388-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskUci3DU/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1781966829091300-388-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:46749
--webserver_interface=127.0.97.62
--webserver_port=0
--builtin_ntp_servers=127.0.97.20:40007
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.97.62:46749
--enable_jwt_token_auth=true
--jwks_url=default.url
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--trusted_certificate_file=/tmp/dist-test-taskUci3DU/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1781966829091300-388-0/testchainca.cert with env {}
W20260620 14:47:54.462002 5864 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 14:47:54.462318 5864 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 14:47:54.462500 5864 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 14:47:54.469738 5864 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 14:47:54.469852 5864 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 14:47:54.469959 5864 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 14:47:54.470060 5864 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 14:47:54.470125 5864 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
W20260620 14:47:54.470244 5864 flags.cc:425] Enabled experimental flag: --enable_jwt_token_auth=true
W20260620 14:47:54.470288 5864 flags.cc:425] Enabled experimental flag: --jwks_url=default.url
I20260620 14:47:54.480315 5864 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:40007
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskUci3DU/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1781966829091300-388-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskUci3DU/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1781966829091300-388-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.97.62:46749
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:46749
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--enable_jwt_token_auth=true
--jwks_url=default.url
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskUci3DU/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1781966829091300-388-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.97.62
--webserver_port=0
--trusted_certificate_file=/tmp/dist-test-taskUci3DU/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1781966829091300-388-0/testchainca.cert
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskUci3DU/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1781966829091300-388-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 13:41:20 UTC on ccycloud.quasar-vdagoi.root.comops.site
build id 4457462
ASAN enabled
I20260620 14:47:54.481698 5864 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 14:47:54.483093 5864 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 14:47:54.494779 5869 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 14:47:54.494800 5870 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 14:47:54.494782 5872 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 14:47:54.495515 5864 server_base.cc:1034] running on GCE node
I20260620 14:47:54.496472 5864 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 14:47:54.497901 5864 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 14:47:54.499147 5864 hybrid_clock.cc:648] HybridClock initialized: now 1781966874499097 us; error 44 us; skew 500 ppm
I20260620 14:47:54.499538 5864 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260620 14:47:54.501842 5864 webserver.cc:460] Webserver started at http://127.0.97.62:42507/ using document root <none> and password file <none>
I20260620 14:47:54.502588 5864 fs_manager.cc:359] Metadata directory not provided
I20260620 14:47:54.502728 5864 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 14:47:54.503062 5864 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 14:47:54.506129 5864 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskUci3DU/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1781966829091300-388-0/minicluster-data/master-0/data/instance:
uuid: "b03f345eeefa424aa152e1fb78d998cd"
format_stamp: "Formatted at 2026-06-20 14:47:54 on dist-test-slave-q9j6"
server_key: "6fee1320597a97c599491705d821d280"
server_key_iv: "a2a46ab0dd5275cbcdaa4869613ca0d7"
server_key_version: "encryptionkey@0"
I20260620 14:47:54.506984 5864 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskUci3DU/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1781966829091300-388-0/minicluster-data/master-0/wal/instance:
uuid: "b03f345eeefa424aa152e1fb78d998cd"
format_stamp: "Formatted at 2026-06-20 14:47:54 on dist-test-slave-q9j6"
server_key: "6fee1320597a97c599491705d821d280"
server_key_iv: "a2a46ab0dd5275cbcdaa4869613ca0d7"
server_key_version: "encryptionkey@0"
I20260620 14:47:54.512390 5864 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.001s sys 0.007s
I20260620 14:47:54.516062 5878 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 14:47:54.517546 5864 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.002s
I20260620 14:47:54.517751 5864 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskUci3DU/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1781966829091300-388-0/minicluster-data/master-0/data,/tmp/dist-test-taskUci3DU/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1781966829091300-388-0/minicluster-data/master-0/wal
uuid: "b03f345eeefa424aa152e1fb78d998cd"
format_stamp: "Formatted at 2026-06-20 14:47:54 on dist-test-slave-q9j6"
server_key: "6fee1320597a97c599491705d821d280"
server_key_iv: "a2a46ab0dd5275cbcdaa4869613ca0d7"
server_key_version: "encryptionkey@0"
I20260620 14:47:54.517992 5864 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskUci3DU/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1781966829091300-388-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskUci3DU/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1781966829091300-388-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskUci3DU/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1781966829091300-388-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 14:47:54.537782 5864 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 14:47:54.538657 5864 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 14:47:54.538954 5864 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 14:47:54.557494 5864 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.62:46749
I20260620 14:47:54.557518 5929 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.62:46749 every 8 connection(s)
I20260620 14:47:54.558880 5864 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskUci3DU/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1781966829091300-388-0/minicluster-data/master-0/data/info.pb
I20260620 14:47:54.562669 5930 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 14:47:54.565572 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskUci3DU/build/asan/bin/kudu as pid 5864
I20260620 14:47:54.565793 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskUci3DU/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1781966829091300-388-0/minicluster-data/master-0/wal/instance
I20260620 14:47:54.566422 388 external_mini_cluster.cc:1428] Setting key 45c4390a7350bdefb3633d2ff20bf8aa
I20260620 14:47:54.576274 5930 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b03f345eeefa424aa152e1fb78d998cd: Bootstrap starting.
I20260620 14:47:54.580180 5930 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P b03f345eeefa424aa152e1fb78d998cd: Neither blocks nor log segments found. Creating new log.
I20260620 14:47:54.581560 5930 log.cc:824] T 00000000000000000000000000000000 P b03f345eeefa424aa152e1fb78d998cd: Log is configured to *not* fsync() on all Append() calls
I20260620 14:47:54.582320 5933 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:47:54.568444 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54782 (local address 127.0.97.62:46749)
0620 14:47:54.569525 (+ 1081us) server_negotiation.cc:206] Beginning negotiation
0620 14:47:54.569539 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0620 14:47:54.569592 (+ 53us) server_negotiation.cc:407] Connection header received
0620 14:47:54.571648 (+ 2056us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:47:54.571689 (+ 41us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:47:54.572395 (+ 706us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:47:54.572881 (+ 486us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:47:54.573496 (+ 615us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:47:54.575493 (+ 1997us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:47:54.576906 (+ 1413us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:47:54.577940 (+ 1034us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:47:54.578169 (+ 229us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:47:54.579269 (+ 1100us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 14:47:54.579279 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 14:47:54.579297 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 14:47:54.580019 (+ 722us) server_negotiation.cc:922] calling sasl_server_start()
0620 14:47:54.580169 (+ 150us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 14:47:54.580198 (+ 29us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 14:47:54.580316 (+ 118us) server_negotiation.cc:1035] Waiting for connection context
0620 14:47:54.580590 (+ 274us) server_negotiation.cc:299] Negotiation successful
0620 14:47:54.580944 (+ 354us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":563,"thread_start_us":339,"threads_started":1}
I20260620 14:47:54.584765 5930 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b03f345eeefa424aa152e1fb78d998cd: No bootstrap required, opened a new log
I20260620 14:47:54.586292 388 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260620 14:47:54.592934 5930 raft_consensus.cc:348] T 00000000000000000000000000000000 P b03f345eeefa424aa152e1fb78d998cd [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b03f345eeefa424aa152e1fb78d998cd" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 46749 } }
I20260620 14:47:54.593331 5930 raft_consensus.cc:374] T 00000000000000000000000000000000 P b03f345eeefa424aa152e1fb78d998cd [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 14:47:54.593439 5930 raft_consensus.cc:729] T 00000000000000000000000000000000 P b03f345eeefa424aa152e1fb78d998cd [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b03f345eeefa424aa152e1fb78d998cd, State: Initialized, Role: FOLLOWER
I20260620 14:47:54.594115 5930 consensus_queue.cc:260] T 00000000000000000000000000000000 P b03f345eeefa424aa152e1fb78d998cd [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b03f345eeefa424aa152e1fb78d998cd" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 46749 } }
I20260620 14:47:54.594349 5930 raft_consensus.cc:388] T 00000000000000000000000000000000 P b03f345eeefa424aa152e1fb78d998cd [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 14:47:54.594499 5930 raft_consensus.cc:482] T 00000000000000000000000000000000 P b03f345eeefa424aa152e1fb78d998cd [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 14:47:54.594691 5930 raft_consensus.cc:3037] T 00000000000000000000000000000000 P b03f345eeefa424aa152e1fb78d998cd [term 0 FOLLOWER]: Advancing to term 1
I20260620 14:47:54.597071 5930 raft_consensus.cc:504] T 00000000000000000000000000000000 P b03f345eeefa424aa152e1fb78d998cd [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b03f345eeefa424aa152e1fb78d998cd" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 46749 } }
I20260620 14:47:54.597618 5930 leader_election.cc:302] T 00000000000000000000000000000000 P b03f345eeefa424aa152e1fb78d998cd [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: b03f345eeefa424aa152e1fb78d998cd; no voters:
I20260620 14:47:54.598213 5930 leader_election.cc:288] T 00000000000000000000000000000000 P b03f345eeefa424aa152e1fb78d998cd [CANDIDATE]: Term 1 election: Requested vote from peers
I20260620 14:47:54.598373 5935 raft_consensus.cc:2781] T 00000000000000000000000000000000 P b03f345eeefa424aa152e1fb78d998cd [term 1 FOLLOWER]: Leader election won for term 1
I20260620 14:47:54.599077 5935 raft_consensus.cc:686] T 00000000000000000000000000000000 P b03f345eeefa424aa152e1fb78d998cd [term 1 LEADER]: Becoming Leader. State: Replica: b03f345eeefa424aa152e1fb78d998cd, State: Running, Role: LEADER
I20260620 14:47:54.599701 5935 consensus_queue.cc:237] T 00000000000000000000000000000000 P b03f345eeefa424aa152e1fb78d998cd [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b03f345eeefa424aa152e1fb78d998cd" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 46749 } }
I20260620 14:47:54.600545 5930 sys_catalog.cc:564] T 00000000000000000000000000000000 P b03f345eeefa424aa152e1fb78d998cd [sys.catalog]: configured and running, proceeding with master startup.
I20260620 14:47:54.602725 5937 sys_catalog.cc:455] T 00000000000000000000000000000000 P b03f345eeefa424aa152e1fb78d998cd [sys.catalog]: SysCatalogTable state changed. Reason: New leader b03f345eeefa424aa152e1fb78d998cd. Latest consensus state: current_term: 1 leader_uuid: "b03f345eeefa424aa152e1fb78d998cd" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b03f345eeefa424aa152e1fb78d998cd" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 46749 } } }
I20260620 14:47:54.603138 5937 sys_catalog.cc:458] T 00000000000000000000000000000000 P b03f345eeefa424aa152e1fb78d998cd [sys.catalog]: This master's current role is: LEADER
I20260620 14:47:54.604012 5936 sys_catalog.cc:455] T 00000000000000000000000000000000 P b03f345eeefa424aa152e1fb78d998cd [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "b03f345eeefa424aa152e1fb78d998cd" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b03f345eeefa424aa152e1fb78d998cd" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 46749 } } }
I20260620 14:47:54.604526 5936 sys_catalog.cc:458] T 00000000000000000000000000000000 P b03f345eeefa424aa152e1fb78d998cd [sys.catalog]: This master's current role is: LEADER
I20260620 14:47:54.604773 5942 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 14:47:54.610584 5942 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 14:47:54.621330 5942 catalog_manager.cc:1347] Generated new cluster ID: a1eed24fb0fc4b49a1aea687201a846e
I20260620 14:47:54.621511 5942 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 14:47:54.648617 5942 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 14:47:54.650602 5942 catalog_manager.cc:1504] Loading token signing keys...
I20260620 14:47:54.661233 5942 catalog_manager.cc:5868] T 00000000000000000000000000000000 P b03f345eeefa424aa152e1fb78d998cd: Generated new TSK 0
I20260620 14:47:54.662156 5942 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260620 14:47:54.744604 5933 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:47:54.676171 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54790 (local address 127.0.97.62:46749)
0620 14:47:54.676390 (+ 219us) server_negotiation.cc:206] Beginning negotiation
0620 14:47:54.676397 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0620 14:47:54.676791 (+ 394us) server_negotiation.cc:407] Connection header received
0620 14:47:54.677027 (+ 236us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:47:54.677033 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:47:54.677136 (+ 103us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:47:54.677333 (+ 197us) server_negotiation.cc:226] Negotiated authn=JWT
0620 14:47:54.677771 (+ 438us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:47:54.678927 (+ 1156us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:47:54.680376 (+ 1449us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:47:54.681098 (+ 722us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:47:54.681236 (+ 138us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:47:54.681710 (+ 474us) server_negotiation.cc:365] Received JWT_EXCHANGE NegotiatePB request
0620 14:47:54.743842 (+ 62132us) server_negotiation.cc:394] Sending RPC error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'default.url': curl error: Couldn't resolve host name: Could not resolve host: default.url
0620 14:47:54.744278 (+ 436us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:54790: Failed to load JWKS: Error downloading JWKS from 'default.url': curl error: Couldn't resolve host name: Could not resolve host: default.url
Metrics: {"server-negotiator.queue_time_us":70}
W20260620 14:47:54.744763 5933 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:54790: Failed to load JWKS: Error downloading JWKS from 'default.url': curl error: Couldn't resolve host name: Could not resolve host: default.url
W20260620 14:47:54.744833 5959 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 14:47:54.676013 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:46749 (local address 127.0.0.1:54790)
0620 14:47:54.676679 (+ 666us) negotiation.cc:107] Waiting for socket to connect
0620 14:47:54.676695 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0620 14:47:54.676827 (+ 132us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 14:47:54.677347 (+ 520us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 14:47:54.677351 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 14:47:54.677359 (+ 8us) client_negotiation.cc:189] Negotiated authn=JWT
0620 14:47:54.677612 (+ 253us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:47:54.677619 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:47:54.679089 (+ 1470us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:47:54.679093 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:47:54.680227 (+ 1134us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:47:54.680239 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:47:54.681231 (+ 992us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:47:54.681282 (+ 51us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:47:54.681403 (+ 121us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:47:54.681438 (+ 35us) client_negotiation.cc:252] Sending JWT_EXCHANGE NegotiatePB request
0620 14:47:54.744147 (+ 62709us) client_negotiation.cc:283] Received error response from server: Runtime error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'default.url': curl error: Couldn't resolve host name: Could not resolve host: default.url
0620 14:47:54.744443 (+ 296us) negotiation.cc:327] Negotiation complete: Runtime error: Client connection negotiation failed: client connection to 127.0.97.62:46749: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'default.url': curl error: Couldn't resolve host name: Could not resolve host: default.url
Metrics: {"client-negotiator.queue_time_us":517,"thread_start_us":356,"threads_started":1}
I20260620 14:47:54.745954 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskUci3DU/build/asan/bin/kudu pid 5864
I20260620 14:47:54.747190 5925 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 14:47:54.846150 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskUci3DU/build/asan/bin/kudu with pid 5864
2026-06-20T14:47:54Z chronyd exiting
[ OK ] SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS (804 ms)
[ RUN ] SecurityITest.TestWorldReadableKeytab
[ OK ] SecurityITest.TestWorldReadableKeytab (418 ms)
[ RUN ] SecurityITest.TestWorldReadablePrivateKey
[ OK ] SecurityITest.TestWorldReadablePrivateKey (1500 ms)
[ RUN ] SecurityITest.TestCorruptKerberosCC
Loading random data
Initializing database '/tmp/dist-test-taskUci3DU/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781966829091300-388-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 20 14:47:56 dist-test-slave-q9j6 krb5kdc[5975](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 20 14:47:56 dist-test-slave-q9j6 krb5kdc[5975](info): set up 2 sockets
Jun 20 14:47:56 dist-test-slave-q9j6 krb5kdc[5975](info): commencing operation
krb5kdc: starting...
W20260620 14:47:57.864472 388 mini_kdc.cc:121] Time spent starting KDC: real 1.065s user 0.000s sys 0.022s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskUci3DU/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781966829091300-388-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskUci3DU/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781966829091300-388-0/krb5kdc/test-user.keytab.
Jun 20 14:47:57 dist-test-slave-q9j6 krb5kdc[5975](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781966877, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-20T14:47:58Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T14:47:58Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.97.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskUci3DU/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781966829091300-388-0/krb5kdc/kudu_127.0.97.62.keytab.
Entry for principal kudu/127.0.97.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskUci3DU/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781966829091300-388-0/krb5kdc/kudu_127.0.97.62.keytab.
I20260620 14:47:58.074968 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskUci3DU/build/asan/bin/kudu
/tmp/dist-test-taskUci3DU/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskUci3DU/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781966829091300-388-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskUci3DU/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781966829091300-388-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskUci3DU/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781966829091300-388-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskUci3DU/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781966829091300-388-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:35153
--webserver_interface=127.0.97.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskUci3DU/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781966829091300-388-0/krb5kdc/kudu_127.0.97.62.keytab
--principal=kudu/127.0.97.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:35391
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.97.62:35153
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskUci3DU/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781966829091300-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskUci3DU/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781966829091300-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskUci3DU/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781966829091300-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 14:47:58.399413 5988 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 14:47:58.399752 5988 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 14:47:58.399924 5988 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 14:47:58.407261 5988 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 14:47:58.407378 5988 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 14:47:58.407474 5988 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 14:47:58.407583 5988 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 14:47:58.407675 5988 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 14:47:58.417737 5988 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:35391
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskUci3DU/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781966829091300-388-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskUci3DU/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781966829091300-388-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.97.62:35153
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskUci3DU/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781966829091300-388-0/krb5kdc/kudu_127.0.97.62.keytab
--principal=kudu/127.0.97.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:35153
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskUci3DU/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781966829091300-388-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.97.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskUci3DU/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781966829091300-388-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 13:41:20 UTC on ccycloud.quasar-vdagoi.root.comops.site
build id 4457462
ASAN enabled
I20260620 14:47:58.419147 5988 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 14:47:58.420540 5988 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 14:47:58.432698 5996 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 14:47:58.432761 5993 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 14:47:58.432736 5994 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 14:47:58.433384 5988 server_base.cc:1034] running on GCE node
I20260620 14:47:58.434929 5988 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 14:47:58.436435 5988 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 14:47:58.437674 5988 hybrid_clock.cc:648] HybridClock initialized: now 1781966878437629 us; error 45 us; skew 500 ppm
I20260620 14:47:58.438038 5988 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 14:47:58 dist-test-slave-q9j6 krb5kdc[5975](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781966878, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 14:47:58.445622 5988 init.cc:377] Logged in from keytab as kudu/127.0.97.62@KRBTEST.COM (short username kudu)
I20260620 14:47:58.447131 5988 webserver.cc:460] Webserver started at http://127.0.97.62:41811/ using document root <none> and password file <none>
I20260620 14:47:58.447911 5988 fs_manager.cc:359] Metadata directory not provided
I20260620 14:47:58.448025 5988 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 14:47:58.448297 5988 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 14:47:58.451437 5988 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskUci3DU/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781966829091300-388-0/minicluster-data/master-0/data/instance:
uuid: "9f8fe0ab9bce491eba1dc4891f7595a7"
format_stamp: "Formatted at 2026-06-20 14:47:58 on dist-test-slave-q9j6"
server_key: "c228222764009d258a5f93e2fb48274b"
server_key_iv: "b485d2c1fc95f86e95e4fc9ceef88058"
server_key_version: "encryptionkey@0"
I20260620 14:47:58.452301 5988 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskUci3DU/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781966829091300-388-0/minicluster-data/master-0/wal/instance:
uuid: "9f8fe0ab9bce491eba1dc4891f7595a7"
format_stamp: "Formatted at 2026-06-20 14:47:58 on dist-test-slave-q9j6"
server_key: "c228222764009d258a5f93e2fb48274b"
server_key_iv: "b485d2c1fc95f86e95e4fc9ceef88058"
server_key_version: "encryptionkey@0"
I20260620 14:47:58.457759 5988 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260620 14:47:58.461365 6003 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 14:47:58.462800 5988 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260620 14:47:58.462965 5988 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskUci3DU/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781966829091300-388-0/minicluster-data/master-0/data,/tmp/dist-test-taskUci3DU/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781966829091300-388-0/minicluster-data/master-0/wal
uuid: "9f8fe0ab9bce491eba1dc4891f7595a7"
format_stamp: "Formatted at 2026-06-20 14:47:58 on dist-test-slave-q9j6"
server_key: "c228222764009d258a5f93e2fb48274b"
server_key_iv: "b485d2c1fc95f86e95e4fc9ceef88058"
server_key_version: "encryptionkey@0"
I20260620 14:47:58.463148 5988 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskUci3DU/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781966829091300-388-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskUci3DU/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781966829091300-388-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskUci3DU/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781966829091300-388-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 14:47:58.487386 5988 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 14:47:58.492246 5988 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 14:47:58.492481 5988 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 14:47:58.510713 5988 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.62:35153
I20260620 14:47:58.510732 6054 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.62:35153 every 8 connection(s)
I20260620 14:47:58.512187 5988 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskUci3DU/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781966829091300-388-0/minicluster-data/master-0/data/info.pb
I20260620 14:47:58.514679 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskUci3DU/build/asan/bin/kudu as pid 5988
I20260620 14:47:58.514917 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskUci3DU/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781966829091300-388-0/minicluster-data/master-0/wal/instance
I20260620 14:47:58.515530 388 external_mini_cluster.cc:1428] Setting key e802080d4e2ab70fa075b9c8d1620d61
I20260620 14:47:58.516108 6055 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
Jun 20 14:47:58 dist-test-slave-q9j6 krb5kdc[5975](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781966877, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260620 14:47:58.530913 6055 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 9f8fe0ab9bce491eba1dc4891f7595a7: Bootstrap starting.
I20260620 14:47:58.535190 6055 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 9f8fe0ab9bce491eba1dc4891f7595a7: Neither blocks nor log segments found. Creating new log.
I20260620 14:47:58.536533 6055 log.cc:824] T 00000000000000000000000000000000 P 9f8fe0ab9bce491eba1dc4891f7595a7: Log is configured to *not* fsync() on all Append() calls
I20260620 14:47:58.539454 6055 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 9f8fe0ab9bce491eba1dc4891f7595a7: No bootstrap required, opened a new log
I20260620 14:47:58.540323 6058 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:47:58.517468 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42562 (local address 127.0.97.62:35153)
0620 14:47:58.518245 (+ 777us) server_negotiation.cc:206] Beginning negotiation
0620 14:47:58.518256 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0620 14:47:58.518297 (+ 41us) server_negotiation.cc:407] Connection header received
0620 14:47:58.520680 (+ 2383us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:47:58.520704 (+ 24us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:47:58.521117 (+ 413us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:47:58.521456 (+ 339us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:47:58.523221 (+ 1765us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:47:58.524582 (+ 1361us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:47:58.526366 (+ 1784us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:47:58.527101 (+ 735us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:47:58.527248 (+ 147us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:47:58.532130 (+ 4882us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 14:47:58.532142 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 14:47:58.532163 (+ 21us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 14:47:58.532213 (+ 50us) server_negotiation.cc:922] calling sasl_server_start()
0620 14:47:58.536086 (+ 3873us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:47:58.536821 (+ 735us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:47:58.536828 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:47:58.536835 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:47:58.537016 (+ 181us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:47:58.537426 (+ 410us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:47:58.537431 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:47:58.537441 (+ 10us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:47:58.537982 (+ 541us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 14:47:58.538278 (+ 296us) server_negotiation.cc:1035] Waiting for connection context
0620 14:47:58.538718 (+ 440us) server_negotiation.cc:299] Negotiation successful
0620 14:47:58.539135 (+ 417us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":573,"thread_start_us":359,"threads_started":1}
I20260620 14:47:58.547583 6055 raft_consensus.cc:348] T 00000000000000000000000000000000 P 9f8fe0ab9bce491eba1dc4891f7595a7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9f8fe0ab9bce491eba1dc4891f7595a7" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 35153 } }
I20260620 14:47:58.548058 6055 raft_consensus.cc:374] T 00000000000000000000000000000000 P 9f8fe0ab9bce491eba1dc4891f7595a7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 14:47:58.548168 6055 raft_consensus.cc:729] T 00000000000000000000000000000000 P 9f8fe0ab9bce491eba1dc4891f7595a7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9f8fe0ab9bce491eba1dc4891f7595a7, State: Initialized, Role: FOLLOWER
I20260620 14:47:58.548832 6055 consensus_queue.cc:260] T 00000000000000000000000000000000 P 9f8fe0ab9bce491eba1dc4891f7595a7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9f8fe0ab9bce491eba1dc4891f7595a7" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 35153 } }
I20260620 14:47:58.549075 6055 raft_consensus.cc:388] T 00000000000000000000000000000000 P 9f8fe0ab9bce491eba1dc4891f7595a7 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 14:47:58.549219 6055 raft_consensus.cc:482] T 00000000000000000000000000000000 P 9f8fe0ab9bce491eba1dc4891f7595a7 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 14:47:58.549391 6055 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 9f8fe0ab9bce491eba1dc4891f7595a7 [term 0 FOLLOWER]: Advancing to term 1
I20260620 14:47:58.551918 6055 raft_consensus.cc:504] T 00000000000000000000000000000000 P 9f8fe0ab9bce491eba1dc4891f7595a7 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9f8fe0ab9bce491eba1dc4891f7595a7" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 35153 } }
I20260620 14:47:58.552476 6055 leader_election.cc:302] T 00000000000000000000000000000000 P 9f8fe0ab9bce491eba1dc4891f7595a7 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 9f8fe0ab9bce491eba1dc4891f7595a7; no voters:
I20260620 14:47:58.553057 6055 leader_election.cc:288] T 00000000000000000000000000000000 P 9f8fe0ab9bce491eba1dc4891f7595a7 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260620 14:47:58.553210 6060 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 9f8fe0ab9bce491eba1dc4891f7595a7 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 14:47:58.553876 6060 raft_consensus.cc:686] T 00000000000000000000000000000000 P 9f8fe0ab9bce491eba1dc4891f7595a7 [term 1 LEADER]: Becoming Leader. State: Replica: 9f8fe0ab9bce491eba1dc4891f7595a7, State: Running, Role: LEADER
I20260620 14:47:58.554405 6060 consensus_queue.cc:237] T 00000000000000000000000000000000 P 9f8fe0ab9bce491eba1dc4891f7595a7 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9f8fe0ab9bce491eba1dc4891f7595a7" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 35153 } }
I20260620 14:47:58.555184 6055 sys_catalog.cc:564] T 00000000000000000000000000000000 P 9f8fe0ab9bce491eba1dc4891f7595a7 [sys.catalog]: configured and running, proceeding with master startup.
I20260620 14:47:58.557302 6062 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9f8fe0ab9bce491eba1dc4891f7595a7 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 9f8fe0ab9bce491eba1dc4891f7595a7. Latest consensus state: current_term: 1 leader_uuid: "9f8fe0ab9bce491eba1dc4891f7595a7" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9f8fe0ab9bce491eba1dc4891f7595a7" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 35153 } } }
I20260620 14:47:58.557713 6062 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9f8fe0ab9bce491eba1dc4891f7595a7 [sys.catalog]: This master's current role is: LEADER
I20260620 14:47:58.558636 6061 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9f8fe0ab9bce491eba1dc4891f7595a7 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "9f8fe0ab9bce491eba1dc4891f7595a7" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9f8fe0ab9bce491eba1dc4891f7595a7" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 35153 } } }
I20260620 14:47:58.559032 6061 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9f8fe0ab9bce491eba1dc4891f7595a7 [sys.catalog]: This master's current role is: LEADER
I20260620 14:47:58.560206 6065 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 14:47:58.566108 6065 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 14:47:58.575171 6065 catalog_manager.cc:1347] Generated new cluster ID: f1de6c90b3a54c8cb13e72808d0a6b6c
I20260620 14:47:58.575309 6065 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 14:47:58.587250 6065 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 14:47:58.588675 6065 catalog_manager.cc:1504] Loading token signing keys...
I20260620 14:47:58.602598 6065 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 9f8fe0ab9bce491eba1dc4891f7595a7: Generated new TSK 0
I20260620 14:47:58.603510 6065 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.0.97.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskUci3DU/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781966829091300-388-0/krb5kdc/kudu_127.0.97.1.keytab.
Entry for principal kudu/127.0.97.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskUci3DU/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781966829091300-388-0/krb5kdc/kudu_127.0.97.1.keytab.
I20260620 14:47:58.665189 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskUci3DU/build/asan/bin/kudu
/tmp/dist-test-taskUci3DU/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskUci3DU/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781966829091300-388-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskUci3DU/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781966829091300-388-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskUci3DU/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781966829091300-388-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskUci3DU/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781966829091300-388-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.1:0
--local_ip_for_outbound_sockets=127.0.97.1
--webserver_interface=127.0.97.1
--webserver_port=0
--tserver_master_addrs=127.0.97.62:35153
--keytab_file=/tmp/dist-test-taskUci3DU/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781966829091300-388-0/krb5kdc/kudu_127.0.97.1.keytab
--principal=kudu/127.0.97.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:35391
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskUci3DU/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781966829091300-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskUci3DU/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781966829091300-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskUci3DU/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781966829091300-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 14:47:59.006453 6081 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 14:47:59.006755 6081 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 14:47:59.006855 6081 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 14:47:59.014241 6081 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 14:47:59.014400 6081 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 14:47:59.014547 6081 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.1
I20260620 14:47:59.025321 6081 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:35391
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskUci3DU/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781966829091300-388-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskUci3DU/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781966829091300-388-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskUci3DU/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781966829091300-388-0/krb5kdc/kudu_127.0.97.1.keytab
--principal=kudu/127.0.97.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskUci3DU/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781966829091300-388-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.97.1
--webserver_port=0
--tserver_master_addrs=127.0.97.62:35153
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.1
--log_dir=/tmp/dist-test-taskUci3DU/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781966829091300-388-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 13:41:20 UTC on ccycloud.quasar-vdagoi.root.comops.site
build id 4457462
ASAN enabled
I20260620 14:47:59.026724 6081 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 14:47:59.028151 6081 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 14:47:59.041863 6089 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 14:47:59.042116 6086 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 14:47:59.041898 6087 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 14:47:59.045387 6081 server_base.cc:1034] running on GCE node
I20260620 14:47:59.046136 6081 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 14:47:59.047159 6081 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 14:47:59.048394 6081 hybrid_clock.cc:648] HybridClock initialized: now 1781966879048342 us; error 51 us; skew 500 ppm
I20260620 14:47:59.048754 6081 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 14:47:59 dist-test-slave-q9j6 krb5kdc[5975](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781966879, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 14:47:59.052692 6081 init.cc:377] Logged in from keytab as kudu/127.0.97.1@KRBTEST.COM (short username kudu)
I20260620 14:47:59.054700 6081 webserver.cc:460] Webserver started at http://127.0.97.1:35877/ using document root <none> and password file <none>
I20260620 14:47:59.055490 6081 fs_manager.cc:359] Metadata directory not provided
I20260620 14:47:59.055653 6081 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 14:47:59.055986 6081 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 14:47:59.059002 6081 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskUci3DU/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781966829091300-388-0/minicluster-data/ts-0/data/instance:
uuid: "879bcff9a8fa4096a9c77b224b065f76"
format_stamp: "Formatted at 2026-06-20 14:47:59 on dist-test-slave-q9j6"
server_key: "d06c22646daba8a39d784e02cc43be9e"
server_key_iv: "706d6e7294d0564bb1cb58120f8404d9"
server_key_version: "encryptionkey@0"
I20260620 14:47:59.059859 6081 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskUci3DU/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781966829091300-388-0/minicluster-data/ts-0/wal/instance:
uuid: "879bcff9a8fa4096a9c77b224b065f76"
format_stamp: "Formatted at 2026-06-20 14:47:59 on dist-test-slave-q9j6"
server_key: "d06c22646daba8a39d784e02cc43be9e"
server_key_iv: "706d6e7294d0564bb1cb58120f8404d9"
server_key_version: "encryptionkey@0"
I20260620 14:47:59.065105 6081 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.001s sys 0.004s
I20260620 14:47:59.068732 6096 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 14:47:59.070111 6081 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.001s
I20260620 14:47:59.070304 6081 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskUci3DU/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781966829091300-388-0/minicluster-data/ts-0/data,/tmp/dist-test-taskUci3DU/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781966829091300-388-0/minicluster-data/ts-0/wal
uuid: "879bcff9a8fa4096a9c77b224b065f76"
format_stamp: "Formatted at 2026-06-20 14:47:59 on dist-test-slave-q9j6"
server_key: "d06c22646daba8a39d784e02cc43be9e"
server_key_iv: "706d6e7294d0564bb1cb58120f8404d9"
server_key_version: "encryptionkey@0"
I20260620 14:47:59.070530 6081 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskUci3DU/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781966829091300-388-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskUci3DU/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781966829091300-388-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskUci3DU/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781966829091300-388-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 14:47:59.099032 6081 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 14:47:59.103500 6081 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 14:47:59.103881 6081 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 14:47:59.104918 6081 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 14:47:59.106369 6081 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 14:47:59.106514 6081 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 14:47:59.106637 6081 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 14:47:59.106683 6081 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 14:47:59.140219 6081 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.1:35267
I20260620 14:47:59.140252 6208 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.1:35267 every 8 connection(s)
I20260620 14:47:59.141646 6081 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskUci3DU/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781966829091300-388-0/minicluster-data/ts-0/data/info.pb
I20260620 14:47:59.147527 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskUci3DU/build/asan/bin/kudu as pid 6081
I20260620 14:47:59.147801 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskUci3DU/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781966829091300-388-0/minicluster-data/ts-0/wal/instance
I20260620 14:47:59.148519 388 external_mini_cluster.cc:1428] Setting key fa46084e47818289b7526428e66994b4
Jun 20 14:47:59 dist-test-slave-q9j6 krb5kdc[5975](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781966879, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.1@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260620 14:47:59.163909 6211 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:47:59.144366 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.1:53475 (local address 127.0.97.62:35153)
0620 14:47:59.144883 (+ 517us) server_negotiation.cc:206] Beginning negotiation
0620 14:47:59.144888 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 14:47:59.145738 (+ 850us) server_negotiation.cc:407] Connection header received
0620 14:47:59.147774 (+ 2036us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:47:59.147782 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:47:59.147882 (+ 100us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:47:59.148013 (+ 131us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:47:59.149926 (+ 1913us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:47:59.151018 (+ 1092us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:47:59.152923 (+ 1905us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:47:59.153653 (+ 730us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:47:59.153812 (+ 159us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:47:59.157749 (+ 3937us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 14:47:59.157754 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 14:47:59.157758 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 14:47:59.157794 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0620 14:47:59.160472 (+ 2678us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:47:59.161372 (+ 900us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:47:59.161377 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:47:59.161379 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:47:59.161511 (+ 132us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:47:59.162012 (+ 501us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:47:59.162017 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:47:59.162019 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:47:59.162289 (+ 270us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 14:47:59.162456 (+ 167us) server_negotiation.cc:1035] Waiting for connection context
0620 14:47:59.163258 (+ 802us) server_negotiation.cc:299] Negotiation successful
0620 14:47:59.163507 (+ 249us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":401,"thread_start_us":291,"threads_started":1}
I20260620 14:47:59.164956 6212 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:47:59.144846 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:35153 (local address 127.0.97.1:53475)
0620 14:47:59.145566 (+ 720us) negotiation.cc:107] Waiting for socket to connect
0620 14:47:59.145607 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0620 14:47:59.147411 (+ 1804us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 14:47:59.148237 (+ 826us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 14:47:59.148246 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 14:47:59.148869 (+ 623us) client_negotiation.cc:189] Negotiated authn=SASL
0620 14:47:59.149736 (+ 867us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:47:59.149768 (+ 32us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:47:59.151202 (+ 1434us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:47:59.151206 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:47:59.152753 (+ 1547us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:47:59.152767 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:47:59.153915 (+ 1148us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:47:59.153920 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:47:59.154148 (+ 228us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:47:59.154958 (+ 810us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 14:47:59.154983 (+ 25us) client_negotiation.cc:656] calling sasl_client_start()
0620 14:47:59.157473 (+ 2490us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 14:47:59.160653 (+ 3180us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 14:47:59.160663 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 14:47:59.160688 (+ 25us) client_negotiation.cc:760] calling sasl_client_step()
0620 14:47:59.161198 (+ 510us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 14:47:59.161632 (+ 434us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 14:47:59.161638 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 14:47:59.161643 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0620 14:47:59.161861 (+ 218us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 14:47:59.162442 (+ 581us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 14:47:59.162451 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 14:47:59.162853 (+ 402us) client_negotiation.cc:769] Sending connection context
0620 14:47:59.163132 (+ 279us) client_negotiation.cc:240] Negotiation successful
0620 14:47:59.163768 (+ 636us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":474,"thread_start_us":286,"threads_started":1}
I20260620 14:47:59.166862 6209 heartbeater.cc:347] Connected to a master server at 127.0.97.62:35153
I20260620 14:47:59.167192 6209 heartbeater.cc:464] Registering TS with master...
I20260620 14:47:59.168077 6209 heartbeater.cc:511] Master 127.0.97.62:35153 requested a full tablet report, sending...
WARNING: no policy specified for kudu/127.0.97.2@KRBTEST.COM; defaulting to no policy
I20260620 14:47:59.170317 6020 ts_manager.cc:194] Registered new tserver with Master: 879bcff9a8fa4096a9c77b224b065f76 (127.0.97.1:35267)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.2@KRBTEST.COM" created.
I20260620 14:47:59.172048 6020 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.97.1@KRBTEST.COM'} at 127.0.97.1:53475
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskUci3DU/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781966829091300-388-0/krb5kdc/kudu_127.0.97.2.keytab.
Entry for principal kudu/127.0.97.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskUci3DU/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781966829091300-388-0/krb5kdc/kudu_127.0.97.2.keytab.
I20260620 14:47:59.194823 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskUci3DU/build/asan/bin/kudu
/tmp/dist-test-taskUci3DU/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskUci3DU/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781966829091300-388-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskUci3DU/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781966829091300-388-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskUci3DU/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781966829091300-388-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskUci3DU/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781966829091300-388-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.2:0
--local_ip_for_outbound_sockets=127.0.97.2
--webserver_interface=127.0.97.2
--webserver_port=0
--tserver_master_addrs=127.0.97.62:35153
--keytab_file=/tmp/dist-test-taskUci3DU/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781966829091300-388-0/krb5kdc/kudu_127.0.97.2.keytab
--principal=kudu/127.0.97.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:35391
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskUci3DU/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781966829091300-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskUci3DU/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781966829091300-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskUci3DU/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781966829091300-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 14:47:59.536557 6215 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 14:47:59.536845 6215 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 14:47:59.536944 6215 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 14:47:59.544358 6215 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 14:47:59.544478 6215 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 14:47:59.544591 6215 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.2
I20260620 14:47:59.554596 6215 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:35391
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskUci3DU/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781966829091300-388-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskUci3DU/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781966829091300-388-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskUci3DU/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781966829091300-388-0/krb5kdc/kudu_127.0.97.2.keytab
--principal=kudu/127.0.97.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskUci3DU/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781966829091300-388-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.97.2
--webserver_port=0
--tserver_master_addrs=127.0.97.62:35153
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.2
--log_dir=/tmp/dist-test-taskUci3DU/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781966829091300-388-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 13:41:20 UTC on ccycloud.quasar-vdagoi.root.comops.site
build id 4457462
ASAN enabled
I20260620 14:47:59.556157 6215 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 14:47:59.557511 6215 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 14:47:59.571039 6223 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 14:47:59.571100 6220 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 14:47:59.571077 6221 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 14:47:59.571883 6215 server_base.cc:1034] running on GCE node
I20260620 14:47:59.572569 6215 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 14:47:59.573616 6215 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 14:47:59.574857 6215 hybrid_clock.cc:648] HybridClock initialized: now 1781966879574805 us; error 66 us; skew 500 ppm
I20260620 14:47:59.575201 6215 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 14:47:59 dist-test-slave-q9j6 krb5kdc[5975](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781966879, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 14:47:59.578473 6215 init.cc:377] Logged in from keytab as kudu/127.0.97.2@KRBTEST.COM (short username kudu)
I20260620 14:47:59.580103 6215 webserver.cc:460] Webserver started at http://127.0.97.2:40049/ using document root <none> and password file <none>
I20260620 14:47:59.580838 6215 fs_manager.cc:359] Metadata directory not provided
I20260620 14:47:59.580991 6215 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 14:47:59.581303 6215 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 14:47:59.584424 6215 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskUci3DU/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781966829091300-388-0/minicluster-data/ts-1/data/instance:
uuid: "c5cc854d4d8049a1804f782aec7c74ad"
format_stamp: "Formatted at 2026-06-20 14:47:59 on dist-test-slave-q9j6"
server_key: "1cd79beaf549f1e8b5d9ebb53b5aad72"
server_key_iv: "9a08424690740bbb1d065aade0968e1b"
server_key_version: "encryptionkey@0"
I20260620 14:47:59.585240 6215 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskUci3DU/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781966829091300-388-0/minicluster-data/ts-1/wal/instance:
uuid: "c5cc854d4d8049a1804f782aec7c74ad"
format_stamp: "Formatted at 2026-06-20 14:47:59 on dist-test-slave-q9j6"
server_key: "1cd79beaf549f1e8b5d9ebb53b5aad72"
server_key_iv: "9a08424690740bbb1d065aade0968e1b"
server_key_version: "encryptionkey@0"
I20260620 14:47:59.590674 6215 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.000s
I20260620 14:47:59.594344 6230 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 14:47:59.595789 6215 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.002s
I20260620 14:47:59.595981 6215 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskUci3DU/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781966829091300-388-0/minicluster-data/ts-1/data,/tmp/dist-test-taskUci3DU/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781966829091300-388-0/minicluster-data/ts-1/wal
uuid: "c5cc854d4d8049a1804f782aec7c74ad"
format_stamp: "Formatted at 2026-06-20 14:47:59 on dist-test-slave-q9j6"
server_key: "1cd79beaf549f1e8b5d9ebb53b5aad72"
server_key_iv: "9a08424690740bbb1d065aade0968e1b"
server_key_version: "encryptionkey@0"
I20260620 14:47:59.596194 6215 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskUci3DU/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781966829091300-388-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskUci3DU/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781966829091300-388-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskUci3DU/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781966829091300-388-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 14:47:59.625227 6215 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 14:47:59.629477 6215 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 14:47:59.629814 6215 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 14:47:59.630849 6215 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 14:47:59.632280 6215 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 14:47:59.632396 6215 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 14:47:59.632558 6215 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 14:47:59.632633 6215 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 14:47:59.666432 6215 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.2:44277
I20260620 14:47:59.666467 6342 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.2:44277 every 8 connection(s)
I20260620 14:47:59.668004 6215 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskUci3DU/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781966829091300-388-0/minicluster-data/ts-1/data/info.pb
I20260620 14:47:59.676091 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskUci3DU/build/asan/bin/kudu as pid 6215
I20260620 14:47:59.676301 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskUci3DU/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781966829091300-388-0/minicluster-data/ts-1/wal/instance
I20260620 14:47:59.676786 388 external_mini_cluster.cc:1428] Setting key 36fdb1c0df63dbc29ff3c19f11708758
Jun 20 14:47:59 dist-test-slave-q9j6 krb5kdc[5975](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781966879, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.2@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260620 14:47:59.688563 6345 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:47:59.670611 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.2:57507 (local address 127.0.97.62:35153)
0620 14:47:59.671005 (+ 394us) server_negotiation.cc:206] Beginning negotiation
0620 14:47:59.671010 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 14:47:59.672151 (+ 1141us) server_negotiation.cc:407] Connection header received
0620 14:47:59.673945 (+ 1794us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:47:59.673950 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:47:59.674018 (+ 68us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:47:59.674115 (+ 97us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:47:59.675745 (+ 1630us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:47:59.676533 (+ 788us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:47:59.678619 (+ 2086us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:47:59.679015 (+ 396us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:47:59.679164 (+ 149us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:47:59.682643 (+ 3479us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 14:47:59.682648 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 14:47:59.682652 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 14:47:59.682684 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0620 14:47:59.685379 (+ 2695us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:47:59.686167 (+ 788us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:47:59.686172 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:47:59.686175 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:47:59.686295 (+ 120us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:47:59.686747 (+ 452us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:47:59.686759 (+ 12us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:47:59.686761 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:47:59.687017 (+ 256us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 14:47:59.687180 (+ 163us) server_negotiation.cc:1035] Waiting for connection context
0620 14:47:59.687968 (+ 788us) server_negotiation.cc:299] Negotiation successful
0620 14:47:59.688193 (+ 225us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":286,"thread_start_us":190,"threads_started":1}
I20260620 14:47:59.689613 6346 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:47:59.671224 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:35153 (local address 127.0.97.2:57507)
0620 14:47:59.671976 (+ 752us) negotiation.cc:107] Waiting for socket to connect
0620 14:47:59.672020 (+ 44us) client_negotiation.cc:174] Beginning negotiation
0620 14:47:59.673628 (+ 1608us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 14:47:59.674326 (+ 698us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 14:47:59.674334 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 14:47:59.674949 (+ 615us) client_negotiation.cc:189] Negotiated authn=SASL
0620 14:47:59.675526 (+ 577us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:47:59.675544 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:47:59.676677 (+ 1133us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:47:59.676682 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:47:59.678447 (+ 1765us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:47:59.678461 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:47:59.679243 (+ 782us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:47:59.679250 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:47:59.679487 (+ 237us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:47:59.680213 (+ 726us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 14:47:59.680236 (+ 23us) client_negotiation.cc:656] calling sasl_client_start()
0620 14:47:59.682379 (+ 2143us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 14:47:59.685544 (+ 3165us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 14:47:59.685553 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 14:47:59.685573 (+ 20us) client_negotiation.cc:760] calling sasl_client_step()
0620 14:47:59.686023 (+ 450us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 14:47:59.686411 (+ 388us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 14:47:59.686416 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 14:47:59.686420 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0620 14:47:59.686624 (+ 204us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 14:47:59.687154 (+ 530us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 14:47:59.687163 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 14:47:59.687516 (+ 353us) client_negotiation.cc:769] Sending connection context
0620 14:47:59.687856 (+ 340us) client_negotiation.cc:240] Negotiation successful
0620 14:47:59.688434 (+ 578us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":521,"thread_start_us":349,"threads_started":1}
I20260620 14:47:59.691220 6343 heartbeater.cc:347] Connected to a master server at 127.0.97.62:35153
I20260620 14:47:59.691535 6343 heartbeater.cc:464] Registering TS with master...
I20260620 14:47:59.692324 6343 heartbeater.cc:511] Master 127.0.97.62:35153 requested a full tablet report, sending...
I20260620 14:47:59.693857 6020 ts_manager.cc:194] Registered new tserver with Master: c5cc854d4d8049a1804f782aec7c74ad (127.0.97.2:44277)
I20260620 14:47:59.694669 6020 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.97.2@KRBTEST.COM'} at 127.0.97.2:57507
WARNING: no policy specified for kudu/127.0.97.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskUci3DU/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781966829091300-388-0/krb5kdc/kudu_127.0.97.3.keytab.
Entry for principal kudu/127.0.97.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskUci3DU/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781966829091300-388-0/krb5kdc/kudu_127.0.97.3.keytab.
I20260620 14:47:59.722153 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskUci3DU/build/asan/bin/kudu
/tmp/dist-test-taskUci3DU/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskUci3DU/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781966829091300-388-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskUci3DU/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781966829091300-388-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskUci3DU/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781966829091300-388-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskUci3DU/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781966829091300-388-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.3:0
--local_ip_for_outbound_sockets=127.0.97.3
--webserver_interface=127.0.97.3
--webserver_port=0
--tserver_master_addrs=127.0.97.62:35153
--keytab_file=/tmp/dist-test-taskUci3DU/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781966829091300-388-0/krb5kdc/kudu_127.0.97.3.keytab
--principal=kudu/127.0.97.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:35391
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskUci3DU/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781966829091300-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskUci3DU/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781966829091300-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskUci3DU/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781966829091300-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 14:48:00.052556 6349 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 14:48:00.052997 6349 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 14:48:00.053162 6349 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 14:48:00.062251 6349 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 14:48:00.062428 6349 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 14:48:00.062638 6349 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.3
I20260620 14:48:00.072897 6349 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:35391
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskUci3DU/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781966829091300-388-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskUci3DU/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781966829091300-388-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskUci3DU/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781966829091300-388-0/krb5kdc/kudu_127.0.97.3.keytab
--principal=kudu/127.0.97.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskUci3DU/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781966829091300-388-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.97.3
--webserver_port=0
--tserver_master_addrs=127.0.97.62:35153
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.3
--log_dir=/tmp/dist-test-taskUci3DU/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781966829091300-388-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 13:41:20 UTC on ccycloud.quasar-vdagoi.root.comops.site
build id 4457462
ASAN enabled
I20260620 14:48:00.074257 6349 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 14:48:00.075717 6349 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 14:48:00.089558 6354 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 14:48:00.089527 6357 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 14:48:00.089593 6355 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 14:48:00.090248 6349 server_base.cc:1034] running on GCE node
I20260620 14:48:00.090930 6349 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 14:48:00.092041 6349 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 14:48:00.093319 6349 hybrid_clock.cc:648] HybridClock initialized: now 1781966880093242 us; error 73 us; skew 500 ppm
I20260620 14:48:00.093712 6349 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 14:48:00 dist-test-slave-q9j6 krb5kdc[5975](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781966880, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 14:48:00.097180 6349 init.cc:377] Logged in from keytab as kudu/127.0.97.3@KRBTEST.COM (short username kudu)
I20260620 14:48:00.098994 6349 webserver.cc:460] Webserver started at http://127.0.97.3:36797/ using document root <none> and password file <none>
I20260620 14:48:00.100060 6349 fs_manager.cc:359] Metadata directory not provided
I20260620 14:48:00.100212 6349 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 14:48:00.100605 6349 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 14:48:00.104527 6349 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskUci3DU/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781966829091300-388-0/minicluster-data/ts-2/data/instance:
uuid: "31771da9afe64eee8b8485f0d8f8137c"
format_stamp: "Formatted at 2026-06-20 14:48:00 on dist-test-slave-q9j6"
server_key: "2154bd11315079cb04affa4b4e750835"
server_key_iv: "08e50851f426c36614dbffa98f1808bb"
server_key_version: "encryptionkey@0"
I20260620 14:48:00.105794 6349 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskUci3DU/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781966829091300-388-0/minicluster-data/ts-2/wal/instance:
uuid: "31771da9afe64eee8b8485f0d8f8137c"
format_stamp: "Formatted at 2026-06-20 14:48:00 on dist-test-slave-q9j6"
server_key: "2154bd11315079cb04affa4b4e750835"
server_key_iv: "08e50851f426c36614dbffa98f1808bb"
server_key_version: "encryptionkey@0"
I20260620 14:48:00.113335 6349 fs_manager.cc:691] Time spent creating directory manager: real 0.007s user 0.008s sys 0.000s
I20260620 14:48:00.117362 6364 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 14:48:00.118965 6349 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260620 14:48:00.119127 6349 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskUci3DU/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781966829091300-388-0/minicluster-data/ts-2/data,/tmp/dist-test-taskUci3DU/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781966829091300-388-0/minicluster-data/ts-2/wal
uuid: "31771da9afe64eee8b8485f0d8f8137c"
format_stamp: "Formatted at 2026-06-20 14:48:00 on dist-test-slave-q9j6"
server_key: "2154bd11315079cb04affa4b4e750835"
server_key_iv: "08e50851f426c36614dbffa98f1808bb"
server_key_version: "encryptionkey@0"
I20260620 14:48:00.119318 6349 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskUci3DU/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781966829091300-388-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskUci3DU/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781966829091300-388-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskUci3DU/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781966829091300-388-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 14:48:00.145396 6349 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 14:48:00.149516 6349 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 14:48:00.149765 6349 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 14:48:00.150676 6349 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 14:48:00.152045 6349 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 14:48:00.152119 6349 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 14:48:00.152204 6349 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 14:48:00.152248 6349 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 14:48:00.175362 6209 heartbeater.cc:503] Master 127.0.97.62:35153 was elected leader, sending a full tablet report...
I20260620 14:48:00.189045 6349 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.3:44227
I20260620 14:48:00.189241 6476 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.3:44227 every 8 connection(s)
I20260620 14:48:00.190361 6349 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskUci3DU/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781966829091300-388-0/minicluster-data/ts-2/data/info.pb
I20260620 14:48:00.192637 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskUci3DU/build/asan/bin/kudu as pid 6349
I20260620 14:48:00.192833 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskUci3DU/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781966829091300-388-0/minicluster-data/ts-2/wal/instance
I20260620 14:48:00.193230 388 external_mini_cluster.cc:1428] Setting key 0b7e973b1b7a53e12e85d061645f221f
Jun 20 14:48:00 dist-test-slave-q9j6 krb5kdc[5975](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781966880, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.3@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260620 14:48:00.211216 6480 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:48:00.194176 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.3:49243 (local address 127.0.97.62:35153)
0620 14:48:00.195959 (+ 1783us) server_negotiation.cc:206] Beginning negotiation
0620 14:48:00.195963 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 14:48:00.195979 (+ 16us) server_negotiation.cc:407] Connection header received
0620 14:48:00.197085 (+ 1106us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:48:00.197089 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:48:00.197155 (+ 66us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:48:00.197247 (+ 92us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:48:00.198869 (+ 1622us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:48:00.199727 (+ 858us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:48:00.201374 (+ 1647us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:48:00.201764 (+ 390us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:48:00.201870 (+ 106us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:48:00.205294 (+ 3424us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 14:48:00.205298 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 14:48:00.205302 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 14:48:00.205336 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0620 14:48:00.208092 (+ 2756us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:48:00.208864 (+ 772us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:48:00.208871 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:48:00.208875 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:48:00.209000 (+ 125us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:48:00.209443 (+ 443us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:48:00.209450 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:48:00.209453 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:48:00.209725 (+ 272us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 14:48:00.209885 (+ 160us) server_negotiation.cc:1035] Waiting for connection context
0620 14:48:00.210617 (+ 732us) server_negotiation.cc:299] Negotiation successful
0620 14:48:00.210836 (+ 219us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1681,"thread_start_us":1588,"threads_started":1}
I20260620 14:48:00.212304 6479 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:48:00.193687 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:35153 (local address 127.0.97.3:49243)
0620 14:48:00.194525 (+ 838us) negotiation.cc:107] Waiting for socket to connect
0620 14:48:00.194596 (+ 71us) client_negotiation.cc:174] Beginning negotiation
0620 14:48:00.196817 (+ 2221us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 14:48:00.197470 (+ 653us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 14:48:00.197479 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 14:48:00.198062 (+ 583us) client_negotiation.cc:189] Negotiated authn=SASL
0620 14:48:00.198666 (+ 604us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:48:00.198710 (+ 44us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:48:00.199920 (+ 1210us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:48:00.199924 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:48:00.201244 (+ 1320us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:48:00.201257 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:48:00.201950 (+ 693us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:48:00.201954 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:48:00.202154 (+ 200us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:48:00.202880 (+ 726us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 14:48:00.202903 (+ 23us) client_negotiation.cc:656] calling sasl_client_start()
0620 14:48:00.205085 (+ 2182us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 14:48:00.208249 (+ 3164us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 14:48:00.208259 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 14:48:00.208277 (+ 18us) client_negotiation.cc:760] calling sasl_client_step()
0620 14:48:00.208727 (+ 450us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 14:48:00.209124 (+ 397us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 14:48:00.209129 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 14:48:00.209133 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0620 14:48:00.209329 (+ 196us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 14:48:00.209867 (+ 538us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 14:48:00.209876 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 14:48:00.210237 (+ 361us) client_negotiation.cc:769] Sending connection context
0620 14:48:00.210521 (+ 284us) client_negotiation.cc:240] Negotiation successful
0620 14:48:00.211084 (+ 563us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":525,"thread_start_us":316,"threads_started":1}
I20260620 14:48:00.213622 6477 heartbeater.cc:347] Connected to a master server at 127.0.97.62:35153
I20260620 14:48:00.213940 6477 heartbeater.cc:464] Registering TS with master...
I20260620 14:48:00.214679 6477 heartbeater.cc:511] Master 127.0.97.62:35153 requested a full tablet report, sending...
I20260620 14:48:00.216114 6020 ts_manager.cc:194] Registered new tserver with Master: 31771da9afe64eee8b8485f0d8f8137c (127.0.97.3:44227)
I20260620 14:48:00.216926 6020 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.97.3@KRBTEST.COM'} at 127.0.97.3:49243
I20260620 14:48:00.221197 388 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal test-admin with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskUci3DU/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781966829091300-388-0/krb5kdc/test-admin.keytab.
Entry for principal test-admin with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskUci3DU/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781966829091300-388-0/krb5kdc/test-admin.keytab.
Jun 20 14:48:00 dist-test-slave-q9j6 krb5kdc[5975](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781966880, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 14:48:00.248265 388 init.cc:377] Logged in from keytab as test-admin@KRBTEST.COM (short username test-admin)
W20260620 14:48:00.253767 6485 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 14:48:00.251143 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49048 (local address 127.0.97.1:35267)
0620 14:48:00.251818 (+ 675us) server_negotiation.cc:206] Beginning negotiation
0620 14:48:00.251826 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0620 14:48:00.251853 (+ 27us) server_negotiation.cc:407] Connection header received
0620 14:48:00.251914 (+ 61us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:48:00.251920 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:48:00.252150 (+ 230us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:48:00.252291 (+ 141us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:48:00.253323 (+ 1032us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:49048: BlockingRecv error: recv got EOF from 127.0.0.1:49048 (error 108)
Metrics: {"server-negotiator.queue_time_us":520,"thread_start_us":303,"threads_started":1}
W20260620 14:48:00.254081 388 init.cc:300] Could not find kerberos principal in credential cache '/tmp/dist-test-taskUci3DU/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781966829091300-388-0/krb5kdc/krb5cc' of type FILE
Jun 20 14:48:00 dist-test-slave-q9j6 krb5kdc[5975](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781966880, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 14:48:00.255904 388 init.cc:377] Logged in from keytab as test-admin@KRBTEST.COM (short username test-admin)
Jun 20 14:48:00 dist-test-slave-q9j6 krb5kdc[5975](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781966880, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 14:48:00.257450 388 init.cc:295] Successfully reacquired a new kerberos TGT
Jun 20 14:48:00 dist-test-slave-q9j6 krb5kdc[5975](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781966880, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.1@KRBTEST.COM
I20260620 14:48:00.273273 6485 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:48:00.259283 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49062 (local address 127.0.97.1:35267)
0620 14:48:00.259499 (+ 216us) server_negotiation.cc:206] Beginning negotiation
0620 14:48:00.259506 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0620 14:48:00.259688 (+ 182us) server_negotiation.cc:407] Connection header received
0620 14:48:00.259877 (+ 189us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:48:00.259881 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:48:00.259942 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:48:00.260030 (+ 88us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:48:00.261126 (+ 1096us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:48:00.262197 (+ 1071us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:48:00.263445 (+ 1248us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:48:00.264108 (+ 663us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:48:00.264213 (+ 105us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:48:00.267943 (+ 3730us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 14:48:00.267958 (+ 15us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 14:48:00.267974 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 14:48:00.268016 (+ 42us) server_negotiation.cc:922] calling sasl_server_start()
0620 14:48:00.270707 (+ 2691us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:48:00.271408 (+ 701us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:48:00.271414 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:48:00.271416 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:48:00.271538 (+ 122us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:48:00.271934 (+ 396us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:48:00.271937 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:48:00.271939 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:48:00.272296 (+ 357us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 14:48:00.272512 (+ 216us) server_negotiation.cc:1035] Waiting for connection context
0620 14:48:00.272799 (+ 287us) server_negotiation.cc:299] Negotiation successful
0620 14:48:00.273001 (+ 202us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":80}
W20260620 14:48:00.279156 6485 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 14:48:00.277543 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49068 (local address 127.0.97.1:35267)
0620 14:48:00.277779 (+ 236us) server_negotiation.cc:206] Beginning negotiation
0620 14:48:00.277785 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 14:48:00.277953 (+ 168us) server_negotiation.cc:407] Connection header received
0620 14:48:00.278083 (+ 130us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:48:00.278086 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:48:00.278147 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:48:00.278225 (+ 78us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:48:00.278898 (+ 673us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:49068: BlockingRecv error: recv got EOF from 127.0.0.1:49068 (error 108)
Metrics: {"server-negotiator.queue_time_us":92}
W20260620 14:48:00.279285 388 init.cc:300] Could not find kerberos principal in credential cache '/tmp/dist-test-taskUci3DU/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781966829091300-388-0/krb5kdc/krb5cc' of type FILE
Jun 20 14:48:00 dist-test-slave-q9j6 krb5kdc[5975](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781966880, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 14:48:00.280905 388 init.cc:377] Logged in from keytab as test-admin@KRBTEST.COM (short username test-admin)
Jun 20 14:48:00 dist-test-slave-q9j6 krb5kdc[5975](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781966880, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 14:48:00.282200 388 init.cc:295] Successfully reacquired a new kerberos TGT
Jun 20 14:48:00 dist-test-slave-q9j6 krb5kdc[5975](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781966880, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.1@KRBTEST.COM
I20260620 14:48:00.296667 6485 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:48:00.283815 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49078 (local address 127.0.97.1:35267)
0620 14:48:00.284053 (+ 238us) server_negotiation.cc:206] Beginning negotiation
0620 14:48:00.284061 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0620 14:48:00.284331 (+ 270us) server_negotiation.cc:407] Connection header received
0620 14:48:00.284540 (+ 209us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:48:00.284544 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:48:00.284607 (+ 63us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:48:00.284711 (+ 104us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:48:00.285646 (+ 935us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:48:00.286373 (+ 727us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:48:00.287587 (+ 1214us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:48:00.288040 (+ 453us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:48:00.288206 (+ 166us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:48:00.291871 (+ 3665us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 14:48:00.291877 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 14:48:00.291880 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 14:48:00.291919 (+ 39us) server_negotiation.cc:922] calling sasl_server_start()
0620 14:48:00.294293 (+ 2374us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:48:00.294922 (+ 629us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:48:00.294926 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:48:00.294928 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:48:00.295068 (+ 140us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:48:00.295418 (+ 350us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:48:00.295422 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:48:00.295424 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:48:00.295827 (+ 403us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 14:48:00.296030 (+ 203us) server_negotiation.cc:1035] Waiting for connection context
0620 14:48:00.296277 (+ 247us) server_negotiation.cc:299] Negotiation successful
0620 14:48:00.296438 (+ 161us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":83}
W20260620 14:48:00.301016 6485 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 14:48:00.299307 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49092 (local address 127.0.97.1:35267)
0620 14:48:00.299505 (+ 198us) server_negotiation.cc:206] Beginning negotiation
0620 14:48:00.299510 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 14:48:00.299719 (+ 209us) server_negotiation.cc:407] Connection header received
0620 14:48:00.299853 (+ 134us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:48:00.299857 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:48:00.299971 (+ 114us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:48:00.300129 (+ 158us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:48:00.300765 (+ 636us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:49092: BlockingRecv error: recv got EOF from 127.0.0.1:49092 (error 108)
Metrics: {"server-negotiator.queue_time_us":76}
W20260620 14:48:00.301112 388 init.cc:300] Could not find kerberos principal in credential cache '/tmp/dist-test-taskUci3DU/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781966829091300-388-0/krb5kdc/krb5cc' of type FILE
Jun 20 14:48:00 dist-test-slave-q9j6 krb5kdc[5975](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781966880, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 14:48:00.302629 388 init.cc:377] Logged in from keytab as test-admin@KRBTEST.COM (short username test-admin)
Jun 20 14:48:00 dist-test-slave-q9j6 krb5kdc[5975](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781966880, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 14:48:00.303915 388 init.cc:295] Successfully reacquired a new kerberos TGT
Jun 20 14:48:00 dist-test-slave-q9j6 krb5kdc[5975](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781966880, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.1@KRBTEST.COM
I20260620 14:48:00.317484 6485 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:48:00.305304 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49094 (local address 127.0.97.1:35267)
0620 14:48:00.305515 (+ 211us) server_negotiation.cc:206] Beginning negotiation
0620 14:48:00.305520 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 14:48:00.305597 (+ 77us) server_negotiation.cc:407] Connection header received
0620 14:48:00.305721 (+ 124us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:48:00.305725 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:48:00.305780 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:48:00.305859 (+ 79us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:48:00.306782 (+ 923us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:48:00.307490 (+ 708us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:48:00.308694 (+ 1204us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:48:00.309092 (+ 398us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:48:00.309193 (+ 101us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:48:00.312573 (+ 3380us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 14:48:00.312578 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 14:48:00.312582 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 14:48:00.312619 (+ 37us) server_negotiation.cc:922] calling sasl_server_start()
0620 14:48:00.314982 (+ 2363us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:48:00.315594 (+ 612us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:48:00.315653 (+ 59us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:48:00.315661 (+ 8us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:48:00.315810 (+ 149us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:48:00.316142 (+ 332us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:48:00.316146 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:48:00.316148 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:48:00.316394 (+ 246us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 14:48:00.316648 (+ 254us) server_negotiation.cc:1035] Waiting for connection context
0620 14:48:00.316821 (+ 173us) server_negotiation.cc:299] Negotiation successful
0620 14:48:00.316985 (+ 164us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":73}
I20260620 14:48:00.319094 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskUci3DU/build/asan/bin/kudu pid 6081
I20260620 14:48:00.330716 6485 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:48:00.319820 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49106 (local address 127.0.97.1:35267)
0620 14:48:00.320020 (+ 200us) server_negotiation.cc:206] Beginning negotiation
0620 14:48:00.320027 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0620 14:48:00.320342 (+ 315us) server_negotiation.cc:407] Connection header received
0620 14:48:00.320452 (+ 110us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:48:00.320456 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:48:00.320512 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:48:00.320589 (+ 77us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:48:00.321494 (+ 905us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:48:00.322159 (+ 665us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:48:00.323344 (+ 1185us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:48:00.323933 (+ 589us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:48:00.324028 (+ 95us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:48:00.326016 (+ 1988us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 14:48:00.326020 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 14:48:00.326023 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 14:48:00.326053 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0620 14:48:00.328414 (+ 2361us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:48:00.329036 (+ 622us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:48:00.329041 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:48:00.329043 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:48:00.329180 (+ 137us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:48:00.329475 (+ 295us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:48:00.329478 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:48:00.329480 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:48:00.329719 (+ 239us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 14:48:00.329899 (+ 180us) server_negotiation.cc:1035] Waiting for connection context
0620 14:48:00.330184 (+ 285us) server_negotiation.cc:299] Negotiation successful
0620 14:48:00.330348 (+ 164us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":58}
I20260620 14:48:00.331341 6204 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 14:48:00.416088 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskUci3DU/build/asan/bin/kudu with pid 6081
I20260620 14:48:00.433354 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskUci3DU/build/asan/bin/kudu pid 6215
Jun 20 14:48:00 dist-test-slave-q9j6 krb5kdc[5975](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781966880, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.2@KRBTEST.COM
I20260620 14:48:00.449175 6498 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:48:00.434269 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59786 (local address 127.0.97.2:44277)
0620 14:48:00.434815 (+ 546us) server_negotiation.cc:206] Beginning negotiation
0620 14:48:00.434823 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0620 14:48:00.434850 (+ 27us) server_negotiation.cc:407] Connection header received
0620 14:48:00.434912 (+ 62us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:48:00.434919 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:48:00.435153 (+ 234us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:48:00.435287 (+ 134us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:48:00.436367 (+ 1080us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:48:00.437356 (+ 989us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:48:00.438506 (+ 1150us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:48:00.439032 (+ 526us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:48:00.439126 (+ 94us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:48:00.443166 (+ 4040us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 14:48:00.443180 (+ 14us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 14:48:00.443196 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 14:48:00.443240 (+ 44us) server_negotiation.cc:922] calling sasl_server_start()
0620 14:48:00.446291 (+ 3051us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:48:00.447032 (+ 741us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:48:00.447039 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:48:00.447041 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:48:00.447167 (+ 126us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:48:00.447564 (+ 397us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:48:00.447569 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:48:00.447571 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:48:00.448015 (+ 444us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 14:48:00.448271 (+ 256us) server_negotiation.cc:1035] Waiting for connection context
0620 14:48:00.448561 (+ 290us) server_negotiation.cc:299] Negotiation successful
0620 14:48:00.448841 (+ 280us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":393,"thread_start_us":275,"threads_started":1}
I20260620 14:48:00.450457 6338 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 14:48:00.535363 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskUci3DU/build/asan/bin/kudu with pid 6215
I20260620 14:48:00.551116 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskUci3DU/build/asan/bin/kudu pid 6349
Jun 20 14:48:00 dist-test-slave-q9j6 krb5kdc[5975](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781966880, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.3@KRBTEST.COM
I20260620 14:48:00.567355 6500 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:48:00.552208 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42640 (local address 127.0.97.3:44227)
0620 14:48:00.552824 (+ 616us) server_negotiation.cc:206] Beginning negotiation
0620 14:48:00.552831 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0620 14:48:00.552860 (+ 29us) server_negotiation.cc:407] Connection header received
0620 14:48:00.552917 (+ 57us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:48:00.552923 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:48:00.553165 (+ 242us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:48:00.553345 (+ 180us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:48:00.554460 (+ 1115us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:48:00.555558 (+ 1098us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:48:00.556963 (+ 1405us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:48:00.557549 (+ 586us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:48:00.557700 (+ 151us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:48:00.561708 (+ 4008us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 14:48:00.561726 (+ 18us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 14:48:00.561754 (+ 28us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 14:48:00.561795 (+ 41us) server_negotiation.cc:922] calling sasl_server_start()
0620 14:48:00.564475 (+ 2680us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:48:00.565212 (+ 737us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:48:00.565219 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:48:00.565222 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:48:00.565346 (+ 124us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:48:00.565735 (+ 389us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:48:00.565739 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:48:00.565743 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:48:00.566089 (+ 346us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 14:48:00.566416 (+ 327us) server_negotiation.cc:1035] Waiting for connection context
0620 14:48:00.566726 (+ 310us) server_negotiation.cc:299] Negotiation successful
0620 14:48:00.566991 (+ 265us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":465,"thread_start_us":315,"threads_started":1}
I20260620 14:48:00.568794 6472 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 14:48:00.659173 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskUci3DU/build/asan/bin/kudu with pid 6349
I20260620 14:48:00.675076 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskUci3DU/build/asan/bin/kudu pid 5988
I20260620 14:48:00.676326 6050 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 14:48:00.777701 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskUci3DU/build/asan/bin/kudu with pid 5988
2026-06-20T14:48:00Z chronyd exiting
[ OK ] SecurityITest.TestCorruptKerberosCC (4020 ms)
[ RUN ] SecurityITest.TestNonDefaultPrincipalMultipleMaster
Loading random data
Initializing database '/tmp/dist-test-taskUci3DU/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781966829091300-388-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 20 14:48:00 dist-test-slave-q9j6 krb5kdc[6505](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 20 14:48:00 dist-test-slave-q9j6 krb5kdc[6505](info): set up 2 sockets
Jun 20 14:48:00 dist-test-slave-q9j6 krb5kdc[6505](info): commencing operation
krb5kdc: starting...
W20260620 14:48:01.864455 388 mini_kdc.cc:121] Time spent starting KDC: real 1.044s user 0.003s sys 0.019s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskUci3DU/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781966829091300-388-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskUci3DU/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781966829091300-388-0/krb5kdc/test-user.keytab.
Jun 20 14:48:02 dist-test-slave-q9j6 krb5kdc[6505](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781966882, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-20T14:48:02Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T14:48:02Z Disabled control of system clock
WARNING: no policy specified for oryx/127.0.97.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.0.97.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.0.97.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskUci3DU/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781966829091300-388-0/krb5kdc/oryx_127.0.97.62.keytab.
Entry for principal oryx/127.0.97.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskUci3DU/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781966829091300-388-0/krb5kdc/oryx_127.0.97.62.keytab.
I20260620 14:48:02.081147 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskUci3DU/build/asan/bin/kudu
/tmp/dist-test-taskUci3DU/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskUci3DU/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781966829091300-388-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskUci3DU/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781966829091300-388-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskUci3DU/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781966829091300-388-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskUci3DU/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781966829091300-388-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:39515
--webserver_interface=127.0.97.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskUci3DU/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781966829091300-388-0/krb5kdc/oryx_127.0.97.62.keytab
--principal=oryx/127.0.97.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:33997
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.97.62:39515
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskUci3DU/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781966829091300-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskUci3DU/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781966829091300-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskUci3DU/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781966829091300-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 14:48:02.413725 6518 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 14:48:02.414047 6518 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 14:48:02.414206 6518 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 14:48:02.421756 6518 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 14:48:02.421876 6518 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 14:48:02.421976 6518 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 14:48:02.422056 6518 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 14:48:02.422096 6518 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 14:48:02.432857 6518 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:33997
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskUci3DU/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781966829091300-388-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskUci3DU/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781966829091300-388-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.97.62:39515
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskUci3DU/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781966829091300-388-0/krb5kdc/oryx_127.0.97.62.keytab
--principal=oryx/127.0.97.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:39515
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskUci3DU/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781966829091300-388-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.97.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskUci3DU/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781966829091300-388-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 13:41:20 UTC on ccycloud.quasar-vdagoi.root.comops.site
build id 4457462
ASAN enabled
I20260620 14:48:02.434306 6518 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 14:48:02.435747 6518 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 14:48:02.447714 6523 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 14:48:02.447799 6524 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 14:48:02.448081 6526 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 14:48:02.448587 6518 server_base.cc:1034] running on GCE node
I20260620 14:48:02.449501 6518 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 14:48:02.451076 6518 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 14:48:02.452323 6518 hybrid_clock.cc:648] HybridClock initialized: now 1781966882452283 us; error 58 us; skew 500 ppm
I20260620 14:48:02.452704 6518 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 14:48:02 dist-test-slave-q9j6 krb5kdc[6505](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781966882, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.97.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 14:48:02.455968 6518 init.cc:377] Logged in from keytab as oryx/127.0.97.62@KRBTEST.COM (short username oryx)
I20260620 14:48:02.457490 6518 webserver.cc:460] Webserver started at http://127.0.97.62:39331/ using document root <none> and password file <none>
I20260620 14:48:02.458364 6518 fs_manager.cc:359] Metadata directory not provided
I20260620 14:48:02.458516 6518 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 14:48:02.458840 6518 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 14:48:02.462013 6518 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskUci3DU/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781966829091300-388-0/minicluster-data/master-0/data/instance:
uuid: "d663f8b7f54d4fe68ce4af5b87767fed"
format_stamp: "Formatted at 2026-06-20 14:48:02 on dist-test-slave-q9j6"
server_key: "fb1fe3a6c33a1333e601dc560d791e76"
server_key_iv: "30863d60937fc767245e42dfbdc8e2c1"
server_key_version: "encryptionkey@0"
I20260620 14:48:02.462850 6518 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskUci3DU/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781966829091300-388-0/minicluster-data/master-0/wal/instance:
uuid: "d663f8b7f54d4fe68ce4af5b87767fed"
format_stamp: "Formatted at 2026-06-20 14:48:02 on dist-test-slave-q9j6"
server_key: "fb1fe3a6c33a1333e601dc560d791e76"
server_key_iv: "30863d60937fc767245e42dfbdc8e2c1"
server_key_version: "encryptionkey@0"
I20260620 14:48:02.468256 6518 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.001s
I20260620 14:48:02.471899 6533 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 14:48:02.473377 6518 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260620 14:48:02.473590 6518 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskUci3DU/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781966829091300-388-0/minicluster-data/master-0/data,/tmp/dist-test-taskUci3DU/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781966829091300-388-0/minicluster-data/master-0/wal
uuid: "d663f8b7f54d4fe68ce4af5b87767fed"
format_stamp: "Formatted at 2026-06-20 14:48:02 on dist-test-slave-q9j6"
server_key: "fb1fe3a6c33a1333e601dc560d791e76"
server_key_iv: "30863d60937fc767245e42dfbdc8e2c1"
server_key_version: "encryptionkey@0"
I20260620 14:48:02.473814 6518 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskUci3DU/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781966829091300-388-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskUci3DU/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781966829091300-388-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskUci3DU/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781966829091300-388-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 14:48:02.490343 6518 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 14:48:02.495049 6518 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 14:48:02.495363 6518 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 14:48:02.514513 6518 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.62:39515
I20260620 14:48:02.514530 6584 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.62:39515 every 8 connection(s)
I20260620 14:48:02.515995 6518 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskUci3DU/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781966829091300-388-0/minicluster-data/master-0/data/info.pb
I20260620 14:48:02.519920 6585 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 14:48:02.522343 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskUci3DU/build/asan/bin/kudu as pid 6518
I20260620 14:48:02.522591 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskUci3DU/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781966829091300-388-0/minicluster-data/master-0/wal/instance
I20260620 14:48:02.523376 388 external_mini_cluster.cc:1428] Setting key d135c98ce9103919cc2bf67c2753345c
I20260620 14:48:02.532548 6585 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d663f8b7f54d4fe68ce4af5b87767fed: Bootstrap starting.
I20260620 14:48:02.535902 6585 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P d663f8b7f54d4fe68ce4af5b87767fed: Neither blocks nor log segments found. Creating new log.
I20260620 14:48:02.536950 6585 log.cc:824] T 00000000000000000000000000000000 P d663f8b7f54d4fe68ce4af5b87767fed: Log is configured to *not* fsync() on all Append() calls
Jun 20 14:48:02 dist-test-slave-q9j6 krb5kdc[6505](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781966882, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM
I20260620 14:48:02.540423 6585 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d663f8b7f54d4fe68ce4af5b87767fed: No bootstrap required, opened a new log
I20260620 14:48:02.546824 6588 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:48:02.525766 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57890 (local address 127.0.97.62:39515)
0620 14:48:02.526634 (+ 868us) server_negotiation.cc:206] Beginning negotiation
0620 14:48:02.526648 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0620 14:48:02.526700 (+ 52us) server_negotiation.cc:407] Connection header received
0620 14:48:02.528520 (+ 1820us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:48:02.528546 (+ 26us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:48:02.528968 (+ 422us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:48:02.529414 (+ 446us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:48:02.530919 (+ 1505us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:48:02.532363 (+ 1444us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:48:02.533811 (+ 1448us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:48:02.534548 (+ 737us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:48:02.534758 (+ 210us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:48:02.539151 (+ 4393us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 14:48:02.539160 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 14:48:02.539185 (+ 25us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 14:48:02.539220 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0620 14:48:02.542439 (+ 3219us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:48:02.543352 (+ 913us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:48:02.543358 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:48:02.543365 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:48:02.543537 (+ 172us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:48:02.543963 (+ 426us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:48:02.543966 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:48:02.543968 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:48:02.544538 (+ 570us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 14:48:02.544746 (+ 208us) server_negotiation.cc:1035] Waiting for connection context
0620 14:48:02.545237 (+ 491us) server_negotiation.cc:299] Negotiation successful
0620 14:48:02.545671 (+ 434us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":643,"thread_start_us":385,"threads_started":1}
I20260620 14:48:02.549328 6585 raft_consensus.cc:348] T 00000000000000000000000000000000 P d663f8b7f54d4fe68ce4af5b87767fed [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d663f8b7f54d4fe68ce4af5b87767fed" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 39515 } }
I20260620 14:48:02.549733 6585 raft_consensus.cc:374] T 00000000000000000000000000000000 P d663f8b7f54d4fe68ce4af5b87767fed [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 14:48:02.549863 6585 raft_consensus.cc:729] T 00000000000000000000000000000000 P d663f8b7f54d4fe68ce4af5b87767fed [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d663f8b7f54d4fe68ce4af5b87767fed, State: Initialized, Role: FOLLOWER
I20260620 14:48:02.550433 6585 consensus_queue.cc:260] T 00000000000000000000000000000000 P d663f8b7f54d4fe68ce4af5b87767fed [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d663f8b7f54d4fe68ce4af5b87767fed" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 39515 } }
I20260620 14:48:02.550675 6585 raft_consensus.cc:388] T 00000000000000000000000000000000 P d663f8b7f54d4fe68ce4af5b87767fed [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 14:48:02.550817 6585 raft_consensus.cc:482] T 00000000000000000000000000000000 P d663f8b7f54d4fe68ce4af5b87767fed [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 14:48:02.550987 6585 raft_consensus.cc:3037] T 00000000000000000000000000000000 P d663f8b7f54d4fe68ce4af5b87767fed [term 0 FOLLOWER]: Advancing to term 1
I20260620 14:48:02.553241 6585 raft_consensus.cc:504] T 00000000000000000000000000000000 P d663f8b7f54d4fe68ce4af5b87767fed [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d663f8b7f54d4fe68ce4af5b87767fed" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 39515 } }
I20260620 14:48:02.553763 6585 leader_election.cc:302] T 00000000000000000000000000000000 P d663f8b7f54d4fe68ce4af5b87767fed [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: d663f8b7f54d4fe68ce4af5b87767fed; no voters:
I20260620 14:48:02.554359 6585 leader_election.cc:288] T 00000000000000000000000000000000 P d663f8b7f54d4fe68ce4af5b87767fed [CANDIDATE]: Term 1 election: Requested vote from peers
I20260620 14:48:02.554489 6590 raft_consensus.cc:2781] T 00000000000000000000000000000000 P d663f8b7f54d4fe68ce4af5b87767fed [term 1 FOLLOWER]: Leader election won for term 1
I20260620 14:48:02.555246 6590 raft_consensus.cc:686] T 00000000000000000000000000000000 P d663f8b7f54d4fe68ce4af5b87767fed [term 1 LEADER]: Becoming Leader. State: Replica: d663f8b7f54d4fe68ce4af5b87767fed, State: Running, Role: LEADER
I20260620 14:48:02.555858 6590 consensus_queue.cc:237] T 00000000000000000000000000000000 P d663f8b7f54d4fe68ce4af5b87767fed [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d663f8b7f54d4fe68ce4af5b87767fed" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 39515 } }
I20260620 14:48:02.556604 6585 sys_catalog.cc:564] T 00000000000000000000000000000000 P d663f8b7f54d4fe68ce4af5b87767fed [sys.catalog]: configured and running, proceeding with master startup.
I20260620 14:48:02.558763 6592 sys_catalog.cc:455] T 00000000000000000000000000000000 P d663f8b7f54d4fe68ce4af5b87767fed [sys.catalog]: SysCatalogTable state changed. Reason: New leader d663f8b7f54d4fe68ce4af5b87767fed. Latest consensus state: current_term: 1 leader_uuid: "d663f8b7f54d4fe68ce4af5b87767fed" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d663f8b7f54d4fe68ce4af5b87767fed" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 39515 } } }
I20260620 14:48:02.559103 6592 sys_catalog.cc:458] T 00000000000000000000000000000000 P d663f8b7f54d4fe68ce4af5b87767fed [sys.catalog]: This master's current role is: LEADER
I20260620 14:48:02.559820 6591 sys_catalog.cc:455] T 00000000000000000000000000000000 P d663f8b7f54d4fe68ce4af5b87767fed [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "d663f8b7f54d4fe68ce4af5b87767fed" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d663f8b7f54d4fe68ce4af5b87767fed" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 39515 } } }
I20260620 14:48:02.560212 6591 sys_catalog.cc:458] T 00000000000000000000000000000000 P d663f8b7f54d4fe68ce4af5b87767fed [sys.catalog]: This master's current role is: LEADER
I20260620 14:48:02.562139 6595 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 14:48:02.568316 6595 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 14:48:02.577481 6595 catalog_manager.cc:1347] Generated new cluster ID: a48bc503bb2347ef82ed8e77d4e21c70
I20260620 14:48:02.577641 6595 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 14:48:02.586977 6595 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 14:48:02.588898 6595 catalog_manager.cc:1504] Loading token signing keys...
I20260620 14:48:02.597069 6595 catalog_manager.cc:5868] T 00000000000000000000000000000000 P d663f8b7f54d4fe68ce4af5b87767fed: Generated new TSK 0
I20260620 14:48:02.598187 6595 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for oryx/127.0.97.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.0.97.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.0.97.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskUci3DU/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781966829091300-388-0/krb5kdc/oryx_127.0.97.1.keytab.
Entry for principal oryx/127.0.97.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskUci3DU/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781966829091300-388-0/krb5kdc/oryx_127.0.97.1.keytab.
I20260620 14:48:02.648603 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskUci3DU/build/asan/bin/kudu
/tmp/dist-test-taskUci3DU/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskUci3DU/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781966829091300-388-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskUci3DU/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781966829091300-388-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskUci3DU/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781966829091300-388-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskUci3DU/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781966829091300-388-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.1:0
--local_ip_for_outbound_sockets=127.0.97.1
--webserver_interface=127.0.97.1
--webserver_port=0
--tserver_master_addrs=127.0.97.62:39515
--keytab_file=/tmp/dist-test-taskUci3DU/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781966829091300-388-0/krb5kdc/oryx_127.0.97.1.keytab
--principal=oryx/127.0.97.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:33997
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskUci3DU/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781966829091300-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskUci3DU/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781966829091300-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskUci3DU/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781966829091300-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 14:48:02.982441 6611 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 14:48:02.982760 6611 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 14:48:02.982857 6611 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 14:48:02.990195 6611 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 14:48:02.990316 6611 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 14:48:02.990430 6611 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.1
I20260620 14:48:03.000463 6611 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:33997
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskUci3DU/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781966829091300-388-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskUci3DU/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781966829091300-388-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskUci3DU/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781966829091300-388-0/krb5kdc/oryx_127.0.97.1.keytab
--principal=oryx/127.0.97.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskUci3DU/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781966829091300-388-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.97.1
--webserver_port=0
--tserver_master_addrs=127.0.97.62:39515
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.1
--log_dir=/tmp/dist-test-taskUci3DU/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781966829091300-388-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 13:41:20 UTC on ccycloud.quasar-vdagoi.root.comops.site
build id 4457462
ASAN enabled
I20260620 14:48:03.001777 6611 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 14:48:03.003129 6611 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 14:48:03.016852 6616 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 14:48:03.016757 6619 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 14:48:03.016829 6617 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 14:48:03.017516 6611 server_base.cc:1034] running on GCE node
I20260620 14:48:03.018255 6611 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 14:48:03.019270 6611 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 14:48:03.020510 6611 hybrid_clock.cc:648] HybridClock initialized: now 1781966883020423 us; error 91 us; skew 500 ppm
I20260620 14:48:03.020867 6611 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 14:48:03 dist-test-slave-q9j6 krb5kdc[6505](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781966883, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.97.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 14:48:03.024322 6611 init.cc:377] Logged in from keytab as oryx/127.0.97.1@KRBTEST.COM (short username oryx)
I20260620 14:48:03.025892 6611 webserver.cc:460] Webserver started at http://127.0.97.1:43933/ using document root <none> and password file <none>
I20260620 14:48:03.026594 6611 fs_manager.cc:359] Metadata directory not provided
I20260620 14:48:03.026715 6611 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 14:48:03.026971 6611 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 14:48:03.030025 6611 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskUci3DU/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781966829091300-388-0/minicluster-data/ts-0/data/instance:
uuid: "76303ed5f6ac43b7866d8efd0313bdc2"
format_stamp: "Formatted at 2026-06-20 14:48:03 on dist-test-slave-q9j6"
server_key: "10a533175bd5b8fde16371fb855bfd45"
server_key_iv: "c3995e4aa86737054ed2a52b7554a949"
server_key_version: "encryptionkey@0"
I20260620 14:48:03.030784 6611 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskUci3DU/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781966829091300-388-0/minicluster-data/ts-0/wal/instance:
uuid: "76303ed5f6ac43b7866d8efd0313bdc2"
format_stamp: "Formatted at 2026-06-20 14:48:03 on dist-test-slave-q9j6"
server_key: "10a533175bd5b8fde16371fb855bfd45"
server_key_iv: "c3995e4aa86737054ed2a52b7554a949"
server_key_version: "encryptionkey@0"
I20260620 14:48:03.036203 6611 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.007s sys 0.001s
I20260620 14:48:03.039896 6626 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 14:48:03.041354 6611 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260620 14:48:03.041502 6611 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskUci3DU/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781966829091300-388-0/minicluster-data/ts-0/data,/tmp/dist-test-taskUci3DU/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781966829091300-388-0/minicluster-data/ts-0/wal
uuid: "76303ed5f6ac43b7866d8efd0313bdc2"
format_stamp: "Formatted at 2026-06-20 14:48:03 on dist-test-slave-q9j6"
server_key: "10a533175bd5b8fde16371fb855bfd45"
server_key_iv: "c3995e4aa86737054ed2a52b7554a949"
server_key_version: "encryptionkey@0"
I20260620 14:48:03.041692 6611 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskUci3DU/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781966829091300-388-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskUci3DU/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781966829091300-388-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskUci3DU/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781966829091300-388-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 14:48:03.056429 6611 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 14:48:03.061069 6611 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 14:48:03.061326 6611 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 14:48:03.062281 6611 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 14:48:03.063803 6611 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 14:48:03.063876 6611 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 14:48:03.063961 6611 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 14:48:03.064008 6611 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 14:48:03.098878 6611 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.1:39249
I20260620 14:48:03.098917 6738 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.1:39249 every 8 connection(s)
I20260620 14:48:03.100306 6611 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskUci3DU/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781966829091300-388-0/minicluster-data/ts-0/data/info.pb
I20260620 14:48:03.110109 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskUci3DU/build/asan/bin/kudu as pid 6611
I20260620 14:48:03.110348 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskUci3DU/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781966829091300-388-0/minicluster-data/ts-0/wal/instance
I20260620 14:48:03.110972 388 external_mini_cluster.cc:1428] Setting key 3a8f193d71ff92d7cb495bd1af71d76f
Jun 20 14:48:03 dist-test-slave-q9j6 krb5kdc[6505](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781966883, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.97.1@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM
I20260620 14:48:03.121714 6741 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:48:03.102940 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.1:39533 (local address 127.0.97.62:39515)
0620 14:48:03.103385 (+ 445us) server_negotiation.cc:206] Beginning negotiation
0620 14:48:03.103389 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 14:48:03.104531 (+ 1142us) server_negotiation.cc:407] Connection header received
0620 14:48:03.106357 (+ 1826us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:48:03.106363 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:48:03.106458 (+ 95us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:48:03.106582 (+ 124us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:48:03.108270 (+ 1688us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:48:03.109001 (+ 731us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:48:03.110778 (+ 1777us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:48:03.111232 (+ 454us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:48:03.111378 (+ 146us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:48:03.115253 (+ 3875us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 14:48:03.115259 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 14:48:03.115262 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 14:48:03.115297 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0620 14:48:03.118085 (+ 2788us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:48:03.119136 (+ 1051us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:48:03.119140 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:48:03.119143 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:48:03.119289 (+ 146us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:48:03.119876 (+ 587us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:48:03.119880 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:48:03.119881 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:48:03.120149 (+ 268us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 14:48:03.120273 (+ 124us) server_negotiation.cc:1035] Waiting for connection context
0620 14:48:03.121259 (+ 986us) server_negotiation.cc:299] Negotiation successful
0620 14:48:03.121449 (+ 190us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":337,"thread_start_us":223,"threads_started":1}
I20260620 14:48:03.123391 6742 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:48:03.103561 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:39515 (local address 127.0.97.1:39533)
0620 14:48:03.104357 (+ 796us) negotiation.cc:107] Waiting for socket to connect
0620 14:48:03.104405 (+ 48us) client_negotiation.cc:174] Beginning negotiation
0620 14:48:03.106049 (+ 1644us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 14:48:03.106842 (+ 793us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 14:48:03.106851 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 14:48:03.107393 (+ 542us) client_negotiation.cc:189] Negotiated authn=SASL
0620 14:48:03.108071 (+ 678us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:48:03.108088 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:48:03.109194 (+ 1106us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:48:03.109198 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:48:03.110609 (+ 1411us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:48:03.110621 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:48:03.111481 (+ 860us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:48:03.111488 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:48:03.111895 (+ 407us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:48:03.112704 (+ 809us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 14:48:03.112730 (+ 26us) client_negotiation.cc:656] calling sasl_client_start()
0620 14:48:03.114994 (+ 2264us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 14:48:03.118284 (+ 3290us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 14:48:03.118295 (+ 11us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 14:48:03.118320 (+ 25us) client_negotiation.cc:760] calling sasl_client_step()
0620 14:48:03.118966 (+ 646us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 14:48:03.119412 (+ 446us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 14:48:03.119417 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 14:48:03.119422 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0620 14:48:03.119745 (+ 323us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 14:48:03.120294 (+ 549us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 14:48:03.120306 (+ 12us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 14:48:03.120832 (+ 526us) client_negotiation.cc:769] Sending connection context
0620 14:48:03.121152 (+ 320us) client_negotiation.cc:240] Negotiation successful
0620 14:48:03.121794 (+ 642us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":556,"thread_start_us":283,"threads_started":1}
I20260620 14:48:03.125126 6739 heartbeater.cc:347] Connected to a master server at 127.0.97.62:39515
I20260620 14:48:03.125494 6739 heartbeater.cc:464] Registering TS with master...
I20260620 14:48:03.126448 6739 heartbeater.cc:511] Master 127.0.97.62:39515 requested a full tablet report, sending...
I20260620 14:48:03.128863 6550 ts_manager.cc:194] Registered new tserver with Master: 76303ed5f6ac43b7866d8efd0313bdc2 (127.0.97.1:39249)
I20260620 14:48:03.130352 6550 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.0.97.1@KRBTEST.COM'} at 127.0.97.1:39533
WARNING: no policy specified for oryx/127.0.97.2@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.0.97.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.0.97.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskUci3DU/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781966829091300-388-0/krb5kdc/oryx_127.0.97.2.keytab.
Entry for principal oryx/127.0.97.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskUci3DU/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781966829091300-388-0/krb5kdc/oryx_127.0.97.2.keytab.
I20260620 14:48:03.160405 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskUci3DU/build/asan/bin/kudu
/tmp/dist-test-taskUci3DU/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskUci3DU/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781966829091300-388-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskUci3DU/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781966829091300-388-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskUci3DU/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781966829091300-388-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskUci3DU/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781966829091300-388-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.2:0
--local_ip_for_outbound_sockets=127.0.97.2
--webserver_interface=127.0.97.2
--webserver_port=0
--tserver_master_addrs=127.0.97.62:39515
--keytab_file=/tmp/dist-test-taskUci3DU/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781966829091300-388-0/krb5kdc/oryx_127.0.97.2.keytab
--principal=oryx/127.0.97.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:33997
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskUci3DU/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781966829091300-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskUci3DU/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781966829091300-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskUci3DU/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781966829091300-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 14:48:03.492372 6745 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 14:48:03.492676 6745 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 14:48:03.492858 6745 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 14:48:03.500109 6745 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 14:48:03.500267 6745 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 14:48:03.500447 6745 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.2
I20260620 14:48:03.510272 6745 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:33997
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskUci3DU/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781966829091300-388-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskUci3DU/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781966829091300-388-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskUci3DU/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781966829091300-388-0/krb5kdc/oryx_127.0.97.2.keytab
--principal=oryx/127.0.97.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskUci3DU/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781966829091300-388-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.97.2
--webserver_port=0
--tserver_master_addrs=127.0.97.62:39515
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.2
--log_dir=/tmp/dist-test-taskUci3DU/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781966829091300-388-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 13:41:20 UTC on ccycloud.quasar-vdagoi.root.comops.site
build id 4457462
ASAN enabled
I20260620 14:48:03.511675 6745 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 14:48:03.513067 6745 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 14:48:03.526574 6751 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 14:48:03.527192 6753 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 14:48:03.527215 6750 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 14:48:03.527770 6745 server_base.cc:1034] running on GCE node
I20260620 14:48:03.528457 6745 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 14:48:03.529551 6745 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 14:48:03.530843 6745 hybrid_clock.cc:648] HybridClock initialized: now 1781966883530790 us; error 50 us; skew 500 ppm
I20260620 14:48:03.531177 6745 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 14:48:03 dist-test-slave-q9j6 krb5kdc[6505](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781966883, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.97.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 14:48:03.537614 6745 init.cc:377] Logged in from keytab as oryx/127.0.97.2@KRBTEST.COM (short username oryx)
I20260620 14:48:03.539239 6745 webserver.cc:460] Webserver started at http://127.0.97.2:41095/ using document root <none> and password file <none>
I20260620 14:48:03.540076 6745 fs_manager.cc:359] Metadata directory not provided
I20260620 14:48:03.540225 6745 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 14:48:03.540530 6745 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 14:48:03.543589 6745 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskUci3DU/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781966829091300-388-0/minicluster-data/ts-1/data/instance:
uuid: "790f4a4136784937b4bfc2572c01a54b"
format_stamp: "Formatted at 2026-06-20 14:48:03 on dist-test-slave-q9j6"
server_key: "416db6f58c20e48f41471f8b01fcd0d7"
server_key_iv: "98c49a11f1dcbec5107c663c5cd261ca"
server_key_version: "encryptionkey@0"
I20260620 14:48:03.544518 6745 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskUci3DU/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781966829091300-388-0/minicluster-data/ts-1/wal/instance:
uuid: "790f4a4136784937b4bfc2572c01a54b"
format_stamp: "Formatted at 2026-06-20 14:48:03 on dist-test-slave-q9j6"
server_key: "416db6f58c20e48f41471f8b01fcd0d7"
server_key_iv: "98c49a11f1dcbec5107c663c5cd261ca"
server_key_version: "encryptionkey@0"
I20260620 14:48:03.550146 6745 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260620 14:48:03.553880 6760 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 14:48:03.555341 6745 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260620 14:48:03.555538 6745 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskUci3DU/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781966829091300-388-0/minicluster-data/ts-1/data,/tmp/dist-test-taskUci3DU/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781966829091300-388-0/minicluster-data/ts-1/wal
uuid: "790f4a4136784937b4bfc2572c01a54b"
format_stamp: "Formatted at 2026-06-20 14:48:03 on dist-test-slave-q9j6"
server_key: "416db6f58c20e48f41471f8b01fcd0d7"
server_key_iv: "98c49a11f1dcbec5107c663c5cd261ca"
server_key_version: "encryptionkey@0"
I20260620 14:48:03.555821 6745 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskUci3DU/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781966829091300-388-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskUci3DU/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781966829091300-388-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskUci3DU/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781966829091300-388-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 14:48:03.580052 6745 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 14:48:03.584264 6745 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 14:48:03.584586 6745 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 14:48:03.585613 6745 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 14:48:03.587037 6745 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 14:48:03.587143 6745 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 14:48:03.587298 6745 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 14:48:03.587373 6745 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 14:48:03.620927 6745 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.2:36327
I20260620 14:48:03.620975 6872 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.2:36327 every 8 connection(s)
I20260620 14:48:03.622287 6745 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskUci3DU/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781966829091300-388-0/minicluster-data/ts-1/data/info.pb
I20260620 14:48:03.631636 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskUci3DU/build/asan/bin/kudu as pid 6745
I20260620 14:48:03.631860 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskUci3DU/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781966829091300-388-0/minicluster-data/ts-1/wal/instance
I20260620 14:48:03.632385 388 external_mini_cluster.cc:1428] Setting key 6b479cdfa60acea56b6d35a12bd6fafd
Jun 20 14:48:03 dist-test-slave-q9j6 krb5kdc[6505](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781966883, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.97.2@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM
I20260620 14:48:03.644734 6875 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:48:03.624894 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.2:47295 (local address 127.0.97.62:39515)
0620 14:48:03.625346 (+ 452us) server_negotiation.cc:206] Beginning negotiation
0620 14:48:03.625351 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 14:48:03.626406 (+ 1055us) server_negotiation.cc:407] Connection header received
0620 14:48:03.628329 (+ 1923us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:48:03.628334 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:48:03.628404 (+ 70us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:48:03.628502 (+ 98us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:48:03.630003 (+ 1501us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:48:03.630704 (+ 701us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:48:03.632921 (+ 2217us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:48:03.633513 (+ 592us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:48:03.633637 (+ 124us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:48:03.637382 (+ 3745us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 14:48:03.637391 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 14:48:03.637398 (+ 7us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 14:48:03.637448 (+ 50us) server_negotiation.cc:922] calling sasl_server_start()
0620 14:48:03.641085 (+ 3637us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:48:03.642159 (+ 1074us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:48:03.642163 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:48:03.642165 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:48:03.642298 (+ 133us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:48:03.642806 (+ 508us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:48:03.642809 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:48:03.642811 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:48:03.643052 (+ 241us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 14:48:03.643155 (+ 103us) server_negotiation.cc:1035] Waiting for connection context
0620 14:48:03.644160 (+ 1005us) server_negotiation.cc:299] Negotiation successful
0620 14:48:03.644405 (+ 245us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":347,"thread_start_us":234,"threads_started":1}
I20260620 14:48:03.646260 6876 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:48:03.625535 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:39515 (local address 127.0.97.2:47295)
0620 14:48:03.626229 (+ 694us) negotiation.cc:107] Waiting for socket to connect
0620 14:48:03.626266 (+ 37us) client_negotiation.cc:174] Beginning negotiation
0620 14:48:03.628051 (+ 1785us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 14:48:03.628720 (+ 669us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 14:48:03.628729 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 14:48:03.629252 (+ 523us) client_negotiation.cc:189] Negotiated authn=SASL
0620 14:48:03.629835 (+ 583us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:48:03.629851 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:48:03.630840 (+ 989us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:48:03.630844 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:48:03.632774 (+ 1930us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:48:03.632789 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:48:03.633724 (+ 935us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:48:03.633729 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:48:03.633926 (+ 197us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:48:03.634673 (+ 747us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 14:48:03.634700 (+ 27us) client_negotiation.cc:656] calling sasl_client_start()
0620 14:48:03.637129 (+ 2429us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 14:48:03.641266 (+ 4137us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 14:48:03.641279 (+ 13us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 14:48:03.641300 (+ 21us) client_negotiation.cc:760] calling sasl_client_step()
0620 14:48:03.642025 (+ 725us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 14:48:03.642414 (+ 389us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 14:48:03.642422 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 14:48:03.642428 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0620 14:48:03.642694 (+ 266us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 14:48:03.643192 (+ 498us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 14:48:03.643203 (+ 11us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 14:48:03.643716 (+ 513us) client_negotiation.cc:769] Sending connection context
0620 14:48:03.644033 (+ 317us) client_negotiation.cc:240] Negotiation successful
0620 14:48:03.644634 (+ 601us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":455,"thread_start_us":262,"threads_started":1}
I20260620 14:48:03.647899 6873 heartbeater.cc:347] Connected to a master server at 127.0.97.62:39515
I20260620 14:48:03.648217 6873 heartbeater.cc:464] Registering TS with master...
I20260620 14:48:03.648957 6873 heartbeater.cc:511] Master 127.0.97.62:39515 requested a full tablet report, sending...
I20260620 14:48:03.650490 6550 ts_manager.cc:194] Registered new tserver with Master: 790f4a4136784937b4bfc2572c01a54b (127.0.97.2:36327)
I20260620 14:48:03.651401 6550 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.0.97.2@KRBTEST.COM'} at 127.0.97.2:47295
WARNING: no policy specified for oryx/127.0.97.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.0.97.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.0.97.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskUci3DU/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781966829091300-388-0/krb5kdc/oryx_127.0.97.3.keytab.
Entry for principal oryx/127.0.97.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskUci3DU/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781966829091300-388-0/krb5kdc/oryx_127.0.97.3.keytab.
I20260620 14:48:03.682649 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskUci3DU/build/asan/bin/kudu
/tmp/dist-test-taskUci3DU/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskUci3DU/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781966829091300-388-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskUci3DU/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781966829091300-388-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskUci3DU/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781966829091300-388-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskUci3DU/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781966829091300-388-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.3:0
--local_ip_for_outbound_sockets=127.0.97.3
--webserver_interface=127.0.97.3
--webserver_port=0
--tserver_master_addrs=127.0.97.62:39515
--keytab_file=/tmp/dist-test-taskUci3DU/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781966829091300-388-0/krb5kdc/oryx_127.0.97.3.keytab
--principal=oryx/127.0.97.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:33997
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskUci3DU/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781966829091300-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskUci3DU/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781966829091300-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskUci3DU/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781966829091300-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 14:48:04.011389 6879 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 14:48:04.011831 6879 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 14:48:04.012023 6879 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 14:48:04.019781 6879 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 14:48:04.019942 6879 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 14:48:04.020136 6879 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.3
I20260620 14:48:04.030260 6879 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:33997
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskUci3DU/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781966829091300-388-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskUci3DU/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781966829091300-388-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskUci3DU/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781966829091300-388-0/krb5kdc/oryx_127.0.97.3.keytab
--principal=oryx/127.0.97.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskUci3DU/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781966829091300-388-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.97.3
--webserver_port=0
--tserver_master_addrs=127.0.97.62:39515
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.3
--log_dir=/tmp/dist-test-taskUci3DU/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781966829091300-388-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 13:41:20 UTC on ccycloud.quasar-vdagoi.root.comops.site
build id 4457462
ASAN enabled
I20260620 14:48:04.031773 6879 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 14:48:04.033180 6879 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 14:48:04.046761 6884 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 14:48:04.046779 6885 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 14:48:04.047226 6887 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 14:48:04.047876 6879 server_base.cc:1034] running on GCE node
I20260620 14:48:04.048516 6879 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 14:48:04.049526 6879 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 14:48:04.050770 6879 hybrid_clock.cc:648] HybridClock initialized: now 1781966884050701 us; error 63 us; skew 500 ppm
I20260620 14:48:04.051116 6879 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 14:48:04 dist-test-slave-q9j6 krb5kdc[6505](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781966884, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.97.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 14:48:04.054826 6879 init.cc:377] Logged in from keytab as oryx/127.0.97.3@KRBTEST.COM (short username oryx)
I20260620 14:48:04.056380 6879 webserver.cc:460] Webserver started at http://127.0.97.3:34635/ using document root <none> and password file <none>
I20260620 14:48:04.057139 6879 fs_manager.cc:359] Metadata directory not provided
I20260620 14:48:04.057291 6879 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 14:48:04.057628 6879 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 14:48:04.060832 6879 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskUci3DU/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781966829091300-388-0/minicluster-data/ts-2/data/instance:
uuid: "361b2f75b32a48cda1d69fd295208d8d"
format_stamp: "Formatted at 2026-06-20 14:48:04 on dist-test-slave-q9j6"
server_key: "ed5828261852db0a8c4e8a0f8054f519"
server_key_iv: "b801ae422cb0cb97b39aab6d6af7f761"
server_key_version: "encryptionkey@0"
I20260620 14:48:04.061728 6879 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskUci3DU/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781966829091300-388-0/minicluster-data/ts-2/wal/instance:
uuid: "361b2f75b32a48cda1d69fd295208d8d"
format_stamp: "Formatted at 2026-06-20 14:48:04 on dist-test-slave-q9j6"
server_key: "ed5828261852db0a8c4e8a0f8054f519"
server_key_iv: "b801ae422cb0cb97b39aab6d6af7f761"
server_key_version: "encryptionkey@0"
I20260620 14:48:04.067395 6879 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.007s sys 0.000s
I20260620 14:48:04.071630 6894 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 14:48:04.073640 6879 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.004s sys 0.000s
I20260620 14:48:04.073897 6879 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskUci3DU/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781966829091300-388-0/minicluster-data/ts-2/data,/tmp/dist-test-taskUci3DU/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781966829091300-388-0/minicluster-data/ts-2/wal
uuid: "361b2f75b32a48cda1d69fd295208d8d"
format_stamp: "Formatted at 2026-06-20 14:48:04 on dist-test-slave-q9j6"
server_key: "ed5828261852db0a8c4e8a0f8054f519"
server_key_iv: "b801ae422cb0cb97b39aab6d6af7f761"
server_key_version: "encryptionkey@0"
I20260620 14:48:04.074100 6879 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskUci3DU/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781966829091300-388-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskUci3DU/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781966829091300-388-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskUci3DU/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781966829091300-388-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 14:48:04.094393 6879 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 14:48:04.099295 6879 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 14:48:04.099680 6879 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 14:48:04.100723 6879 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 14:48:04.102155 6879 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 14:48:04.102270 6879 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 14:48:04.102465 6879 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 14:48:04.102566 6879 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 14:48:04.134056 6739 heartbeater.cc:503] Master 127.0.97.62:39515 was elected leader, sending a full tablet report...
I20260620 14:48:04.135473 6879 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.3:46083
I20260620 14:48:04.135571 7006 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.3:46083 every 8 connection(s)
I20260620 14:48:04.136842 6879 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskUci3DU/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781966829091300-388-0/minicluster-data/ts-2/data/info.pb
I20260620 14:48:04.143824 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskUci3DU/build/asan/bin/kudu as pid 6879
I20260620 14:48:04.144016 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskUci3DU/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781966829091300-388-0/minicluster-data/ts-2/wal/instance
I20260620 14:48:04.144438 388 external_mini_cluster.cc:1428] Setting key c772020c3278f120a664a025aa7edf33
Jun 20 14:48:04 dist-test-slave-q9j6 krb5kdc[6505](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781966884, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.97.3@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM
I20260620 14:48:04.156742 6875 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:48:04.139367 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.3:55093 (local address 127.0.97.62:39515)
0620 14:48:04.139574 (+ 207us) server_negotiation.cc:206] Beginning negotiation
0620 14:48:04.139581 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0620 14:48:04.140944 (+ 1363us) server_negotiation.cc:407] Connection header received
0620 14:48:04.142705 (+ 1761us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:48:04.142710 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:48:04.142827 (+ 117us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:48:04.142950 (+ 123us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:48:04.144723 (+ 1773us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:48:04.145593 (+ 870us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:48:04.147671 (+ 2078us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:48:04.148068 (+ 397us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:48:04.148166 (+ 98us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:48:04.151489 (+ 3323us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 14:48:04.151494 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 14:48:04.151497 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 14:48:04.151529 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0620 14:48:04.153961 (+ 2432us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:48:04.154633 (+ 672us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:48:04.154636 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:48:04.154639 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:48:04.154759 (+ 120us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:48:04.155129 (+ 370us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:48:04.155132 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:48:04.155134 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:48:04.155385 (+ 251us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 14:48:04.155490 (+ 105us) server_negotiation.cc:1035] Waiting for connection context
0620 14:48:04.156291 (+ 801us) server_negotiation.cc:299] Negotiation successful
0620 14:48:04.156467 (+ 176us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":75}
I20260620 14:48:04.157828 7009 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:48:04.140027 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:39515 (local address 127.0.97.3:55093)
0620 14:48:04.140768 (+ 741us) negotiation.cc:107] Waiting for socket to connect
0620 14:48:04.140816 (+ 48us) client_negotiation.cc:174] Beginning negotiation
0620 14:48:04.142419 (+ 1603us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 14:48:04.143173 (+ 754us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 14:48:04.143181 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 14:48:04.143800 (+ 619us) client_negotiation.cc:189] Negotiated authn=SASL
0620 14:48:04.144522 (+ 722us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:48:04.144552 (+ 30us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:48:04.145752 (+ 1200us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:48:04.145758 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:48:04.147459 (+ 1701us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:48:04.147476 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:48:04.148254 (+ 778us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:48:04.148258 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:48:04.148449 (+ 191us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:48:04.149091 (+ 642us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 14:48:04.149117 (+ 26us) client_negotiation.cc:656] calling sasl_client_start()
0620 14:48:04.151285 (+ 2168us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 14:48:04.154094 (+ 2809us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 14:48:04.154101 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 14:48:04.154115 (+ 14us) client_negotiation.cc:760] calling sasl_client_step()
0620 14:48:04.154532 (+ 417us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 14:48:04.154857 (+ 325us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 14:48:04.154860 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 14:48:04.154863 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0620 14:48:04.155038 (+ 175us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 14:48:04.155485 (+ 447us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 14:48:04.155491 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 14:48:04.155904 (+ 413us) client_negotiation.cc:769] Sending connection context
0620 14:48:04.156146 (+ 242us) client_negotiation.cc:240] Negotiation successful
0620 14:48:04.156546 (+ 400us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":487,"thread_start_us":284,"threads_started":1}
I20260620 14:48:04.159373 7007 heartbeater.cc:347] Connected to a master server at 127.0.97.62:39515
I20260620 14:48:04.159747 7007 heartbeater.cc:464] Registering TS with master...
I20260620 14:48:04.160600 7007 heartbeater.cc:511] Master 127.0.97.62:39515 requested a full tablet report, sending...
I20260620 14:48:04.162298 6549 ts_manager.cc:194] Registered new tserver with Master: 361b2f75b32a48cda1d69fd295208d8d (127.0.97.3:46083)
I20260620 14:48:04.163331 6549 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.0.97.3@KRBTEST.COM'} at 127.0.97.3:55093
I20260620 14:48:04.172535 388 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
WARNING: no policy specified for oryx/127.0.97.61@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.0.97.61@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.0.97.61 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskUci3DU/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781966829091300-388-0/krb5kdc/oryx_127.0.97.61.keytab.
Entry for principal oryx/127.0.97.61 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskUci3DU/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781966829091300-388-0/krb5kdc/oryx_127.0.97.61.keytab.
I20260620 14:48:04.217374 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskUci3DU/build/asan/bin/kudu
/tmp/dist-test-taskUci3DU/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskUci3DU/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781966829091300-388-0/minicluster-data/master-1/wal
--fs_data_dirs=/tmp/dist-test-taskUci3DU/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781966829091300-388-0/minicluster-data/master-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskUci3DU/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781966829091300-388-0/minicluster-data/master-1/logs
--server_dump_info_path=/tmp/dist-test-taskUci3DU/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781966829091300-388-0/minicluster-data/master-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.61:41719
--webserver_interface=127.0.97.61
--webserver_port=0
--keytab_file=/tmp/dist-test-taskUci3DU/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781966829091300-388-0/krb5kdc/oryx_127.0.97.61.keytab
--principal=oryx/127.0.97.61
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:33997
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.97.62:39515,127.0.97.61:41719
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskUci3DU/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781966829091300-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskUci3DU/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781966829091300-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskUci3DU/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781966829091300-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 14:48:04.554989 7012 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 14:48:04.555305 7012 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 14:48:04.555393 7012 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 14:48:04.563545 7012 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 14:48:04.563753 7012 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 14:48:04.563827 7012 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 14:48:04.563884 7012 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 14:48:04.563926 7012 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 14:48:04.574353 7012 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:33997
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskUci3DU/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781966829091300-388-0/minicluster-data/master-1/data
--fs_wal_dir=/tmp/dist-test-taskUci3DU/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781966829091300-388-0/minicluster-data/master-1/wal
--ipki_ca_key_size=768
--master_addresses=127.0.97.62:39515,127.0.97.61:41719
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskUci3DU/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781966829091300-388-0/krb5kdc/oryx_127.0.97.61.keytab
--principal=oryx/127.0.97.61
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.61:41719
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskUci3DU/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781966829091300-388-0/minicluster-data/master-1/data/info.pb
--webserver_interface=127.0.97.61
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskUci3DU/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781966829091300-388-0/minicluster-data/master-1/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 13:41:20 UTC on ccycloud.quasar-vdagoi.root.comops.site
build id 4457462
ASAN enabled
W20260620 14:48:04.574712 7012 master_options.cc:55] Only 2 masters are specified by master_addresses_flag ('127.0.97.62:39515,127.0.97.61:41719'), but minimum of 3 are required to tolerate failures of any one master. It is recommended to use at least 3 masters.
I20260620 14:48:04.575848 7012 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 14:48:04.577209 7012 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 14:48:04.589576 7018 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 14:48:04.589673 7020 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 14:48:04.589634 7017 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 14:48:04.590493 7012 server_base.cc:1034] running on GCE node
I20260620 14:48:04.591408 7012 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 14:48:04.592990 7012 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 14:48:04.594249 7012 hybrid_clock.cc:648] HybridClock initialized: now 1781966884594168 us; error 68 us; skew 500 ppm
I20260620 14:48:04.594615 7012 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 14:48:04 dist-test-slave-q9j6 krb5kdc[6505](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781966884, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.97.61@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 14:48:04.597992 7012 init.cc:377] Logged in from keytab as oryx/127.0.97.61@KRBTEST.COM (short username oryx)
I20260620 14:48:04.599545 7012 webserver.cc:460] Webserver started at http://127.0.97.61:33085/ using document root <none> and password file <none>
I20260620 14:48:04.600418 7012 fs_manager.cc:359] Metadata directory not provided
I20260620 14:48:04.600548 7012 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 14:48:04.600868 7012 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 14:48:04.604004 7012 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskUci3DU/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781966829091300-388-0/minicluster-data/master-1/data/instance:
uuid: "55c0ce5da3694b759b5b3113647c3eaa"
format_stamp: "Formatted at 2026-06-20 14:48:04 on dist-test-slave-q9j6"
server_key: "3d85e1244253c50a1bcdb8d6f9abcd11"
server_key_iv: "2e529241fa53477dbdceb9fa62309930"
server_key_version: "encryptionkey@0"
I20260620 14:48:04.604794 7012 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskUci3DU/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781966829091300-388-0/minicluster-data/master-1/wal/instance:
uuid: "55c0ce5da3694b759b5b3113647c3eaa"
format_stamp: "Formatted at 2026-06-20 14:48:04 on dist-test-slave-q9j6"
server_key: "3d85e1244253c50a1bcdb8d6f9abcd11"
server_key_iv: "2e529241fa53477dbdceb9fa62309930"
server_key_version: "encryptionkey@0"
I20260620 14:48:04.610031 7012 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260620 14:48:04.614293 7027 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 14:48:04.615900 7012 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.003s sys 0.000s
I20260620 14:48:04.616047 7012 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskUci3DU/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781966829091300-388-0/minicluster-data/master-1/data,/tmp/dist-test-taskUci3DU/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781966829091300-388-0/minicluster-data/master-1/wal
uuid: "55c0ce5da3694b759b5b3113647c3eaa"
format_stamp: "Formatted at 2026-06-20 14:48:04 on dist-test-slave-q9j6"
server_key: "3d85e1244253c50a1bcdb8d6f9abcd11"
server_key_iv: "2e529241fa53477dbdceb9fa62309930"
server_key_version: "encryptionkey@0"
I20260620 14:48:04.616233 7012 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskUci3DU/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781966829091300-388-0/minicluster-data/master-1/wal
metadata directory: /tmp/dist-test-taskUci3DU/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781966829091300-388-0/minicluster-data/master-1/wal
1 data directories: /tmp/dist-test-taskUci3DU/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781966829091300-388-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 14:48:04.642347 7012 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 14:48:04.646673 7012 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 14:48:04.646894 7012 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 14:48:04.654337 6873 heartbeater.cc:503] Master 127.0.97.62:39515 was elected leader, sending a full tablet report...
I20260620 14:48:04.666224 7012 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.61:41719
I20260620 14:48:04.666241 7078 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.61:41719 every 8 connection(s)
I20260620 14:48:04.667826 7012 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskUci3DU/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781966829091300-388-0/minicluster-data/master-1/data/info.pb
I20260620 14:48:04.671720 7079 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 14:48:04.678324 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskUci3DU/build/asan/bin/kudu as pid 7012
I20260620 14:48:04.678546 7079 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.97.62" port: 39515 } has no permanent_uuid. Determining permanent_uuid...
Jun 20 14:48:04 dist-test-slave-q9j6 krb5kdc[6505](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781966884, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.97.61@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM
Jun 20 14:48:04 dist-test-slave-q9j6 krb5kdc[6505](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781966882, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.0.97.61@KRBTEST.COM
I20260620 14:48:04.702137 7083 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:48:04.681570 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57892 (local address 127.0.97.62:39515)
0620 14:48:04.682063 (+ 493us) server_negotiation.cc:206] Beginning negotiation
0620 14:48:04.682069 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 14:48:04.682647 (+ 578us) server_negotiation.cc:407] Connection header received
0620 14:48:04.683959 (+ 1312us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:48:04.683963 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:48:04.684025 (+ 62us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:48:04.684115 (+ 90us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:48:04.685873 (+ 1758us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:48:04.687043 (+ 1170us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:48:04.689176 (+ 2133us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:48:04.689798 (+ 622us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:48:04.689929 (+ 131us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:48:04.694291 (+ 4362us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 14:48:04.694299 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 14:48:04.694304 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 14:48:04.694350 (+ 46us) server_negotiation.cc:922] calling sasl_server_start()
0620 14:48:04.697182 (+ 2832us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:48:04.698077 (+ 895us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:48:04.698081 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:48:04.698084 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:48:04.698218 (+ 134us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:48:04.698713 (+ 495us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:48:04.698717 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:48:04.698719 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:48:04.699140 (+ 421us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 14:48:04.699317 (+ 177us) server_negotiation.cc:1035] Waiting for connection context
0620 14:48:04.701338 (+ 2021us) server_negotiation.cc:299] Negotiation successful
0620 14:48:04.701602 (+ 264us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":339,"thread_start_us":216,"threads_started":1}
I20260620 14:48:04.702461 7084 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:48:04.681621 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:39515 (local address 127.0.0.1:57892)
0620 14:48:04.682454 (+ 833us) negotiation.cc:107] Waiting for socket to connect
0620 14:48:04.682506 (+ 52us) client_negotiation.cc:174] Beginning negotiation
0620 14:48:04.683710 (+ 1204us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 14:48:04.684212 (+ 502us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 14:48:04.684219 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 14:48:04.685170 (+ 951us) client_negotiation.cc:189] Negotiated authn=SASL
0620 14:48:04.685672 (+ 502us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:48:04.685699 (+ 27us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:48:04.687200 (+ 1501us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:48:04.687204 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:48:04.689026 (+ 1822us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:48:04.689044 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:48:04.690011 (+ 967us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:48:04.690016 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:48:04.690181 (+ 165us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:48:04.691103 (+ 922us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 14:48:04.691137 (+ 34us) client_negotiation.cc:656] calling sasl_client_start()
0620 14:48:04.694058 (+ 2921us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 14:48:04.697319 (+ 3261us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 14:48:04.697332 (+ 13us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 14:48:04.697350 (+ 18us) client_negotiation.cc:760] calling sasl_client_step()
0620 14:48:04.697955 (+ 605us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 14:48:04.698330 (+ 375us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 14:48:04.698337 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 14:48:04.698346 (+ 9us) client_negotiation.cc:760] calling sasl_client_step()
0620 14:48:04.698590 (+ 244us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 14:48:04.699317 (+ 727us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 14:48:04.699326 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 14:48:04.699872 (+ 546us) client_negotiation.cc:769] Sending connection context
0620 14:48:04.700259 (+ 387us) client_negotiation.cc:240] Negotiation successful
0620 14:48:04.700910 (+ 651us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":568,"mutex_wait_us":45,"spinlock_wait_cycles":21888,"thread_start_us":411,"threads_started":1}
I20260620 14:48:04.703015 7082 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:48:04.680661 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48020 (local address 127.0.97.61:41719)
0620 14:48:04.681405 (+ 744us) server_negotiation.cc:206] Beginning negotiation
0620 14:48:04.681454 (+ 49us) server_negotiation.cc:399] Waiting for connection header
0620 14:48:04.681507 (+ 53us) server_negotiation.cc:407] Connection header received
0620 14:48:04.683710 (+ 2203us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:48:04.683738 (+ 28us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:48:04.684036 (+ 298us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:48:04.684263 (+ 227us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:48:04.686059 (+ 1796us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:48:04.687583 (+ 1524us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:48:04.689298 (+ 1715us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:48:04.690145 (+ 847us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:48:04.690264 (+ 119us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:48:04.695228 (+ 4964us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 14:48:04.695242 (+ 14us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 14:48:04.695264 (+ 22us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 14:48:04.695308 (+ 44us) server_negotiation.cc:922] calling sasl_server_start()
0620 14:48:04.698542 (+ 3234us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:48:04.699332 (+ 790us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:48:04.699339 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:48:04.699342 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:48:04.699488 (+ 146us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:48:04.700033 (+ 545us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:48:04.700044 (+ 11us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:48:04.700048 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:48:04.700562 (+ 514us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 14:48:04.700878 (+ 316us) server_negotiation.cc:1035] Waiting for connection context
0620 14:48:04.702029 (+ 1151us) server_negotiation.cc:299] Negotiation successful
0620 14:48:04.702271 (+ 242us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":35,"server-negotiator.queue_time_us":543,"spinlock_wait_cycles":31488,"thread_start_us":342,"threads_started":1}
I20260620 14:48:04.704890 7079 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.97.61" port: 41719 } has no permanent_uuid. Determining permanent_uuid...
Jun 20 14:48:04 dist-test-slave-q9j6 krb5kdc[6505](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781966884, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.97.61@KRBTEST.COM for oryx/127.0.97.61@KRBTEST.COM
I20260620 14:48:04.721115 7084 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:48:04.705798 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.61:41719 (local address 127.0.0.1:48024)
0620 14:48:04.706026 (+ 228us) negotiation.cc:107] Waiting for socket to connect
0620 14:48:04.706047 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0620 14:48:04.706164 (+ 117us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 14:48:04.706874 (+ 710us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 14:48:04.706881 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 14:48:04.707372 (+ 491us) client_negotiation.cc:189] Negotiated authn=SASL
0620 14:48:04.707587 (+ 215us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:48:04.707662 (+ 75us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:48:04.708825 (+ 1163us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:48:04.708834 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:48:04.709986 (+ 1152us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:48:04.709998 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:48:04.710856 (+ 858us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:48:04.710864 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:48:04.711108 (+ 244us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:48:04.711768 (+ 660us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 14:48:04.711786 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0620 14:48:04.714598 (+ 2812us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 14:48:04.718576 (+ 3978us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 14:48:04.718584 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 14:48:04.718592 (+ 8us) client_negotiation.cc:760] calling sasl_client_step()
0620 14:48:04.718990 (+ 398us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 14:48:04.719443 (+ 453us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 14:48:04.719446 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 14:48:04.719449 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0620 14:48:04.719695 (+ 246us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 14:48:04.720370 (+ 675us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 14:48:04.720373 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 14:48:04.720495 (+ 122us) client_negotiation.cc:769] Sending connection context
0620 14:48:04.720598 (+ 103us) client_negotiation.cc:240] Negotiation successful
0620 14:48:04.720829 (+ 231us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":58}
I20260620 14:48:04.721333 7082 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:48:04.706239 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48024 (local address 127.0.97.61:41719)
0620 14:48:04.706423 (+ 184us) server_negotiation.cc:206] Beginning negotiation
0620 14:48:04.706431 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0620 14:48:04.706452 (+ 21us) server_negotiation.cc:407] Connection header received
0620 14:48:04.706509 (+ 57us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:48:04.706516 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:48:04.706597 (+ 81us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:48:04.706716 (+ 119us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:48:04.707804 (+ 1088us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:48:04.708680 (+ 876us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:48:04.710177 (+ 1497us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:48:04.710699 (+ 522us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:48:04.710804 (+ 105us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:48:04.714829 (+ 4025us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 14:48:04.714836 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 14:48:04.714842 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 14:48:04.714883 (+ 41us) server_negotiation.cc:922] calling sasl_server_start()
0620 14:48:04.718371 (+ 3488us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:48:04.719121 (+ 750us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:48:04.719138 (+ 17us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:48:04.719143 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:48:04.719319 (+ 176us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:48:04.719816 (+ 497us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:48:04.719823 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:48:04.719828 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:48:04.720236 (+ 408us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 14:48:04.720406 (+ 170us) server_negotiation.cc:1035] Waiting for connection context
0620 14:48:04.720689 (+ 283us) server_negotiation.cc:299] Negotiation successful
0620 14:48:04.721022 (+ 333us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":57,"server-negotiator.queue_time_us":53}
I20260620 14:48:04.726266 7079 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 55c0ce5da3694b759b5b3113647c3eaa: Bootstrap starting.
I20260620 14:48:04.729447 7079 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 55c0ce5da3694b759b5b3113647c3eaa: Neither blocks nor log segments found. Creating new log.
I20260620 14:48:04.730494 7079 log.cc:824] T 00000000000000000000000000000000 P 55c0ce5da3694b759b5b3113647c3eaa: Log is configured to *not* fsync() on all Append() calls
I20260620 14:48:04.733260 7079 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 55c0ce5da3694b759b5b3113647c3eaa: No bootstrap required, opened a new log
I20260620 14:48:04.740280 7079 raft_consensus.cc:348] T 00000000000000000000000000000000 P 55c0ce5da3694b759b5b3113647c3eaa [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d663f8b7f54d4fe68ce4af5b87767fed" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 39515 } } peers { permanent_uuid: "55c0ce5da3694b759b5b3113647c3eaa" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 41719 } }
I20260620 14:48:04.740710 7079 raft_consensus.cc:374] T 00000000000000000000000000000000 P 55c0ce5da3694b759b5b3113647c3eaa [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 14:48:04.740835 7079 raft_consensus.cc:729] T 00000000000000000000000000000000 P 55c0ce5da3694b759b5b3113647c3eaa [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 55c0ce5da3694b759b5b3113647c3eaa, State: Initialized, Role: FOLLOWER
I20260620 14:48:04.741496 7079 consensus_queue.cc:260] T 00000000000000000000000000000000 P 55c0ce5da3694b759b5b3113647c3eaa [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d663f8b7f54d4fe68ce4af5b87767fed" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 39515 } } peers { permanent_uuid: "55c0ce5da3694b759b5b3113647c3eaa" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 41719 } }
I20260620 14:48:04.742391 7086 sys_catalog.cc:455] T 00000000000000000000000000000000 P 55c0ce5da3694b759b5b3113647c3eaa [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d663f8b7f54d4fe68ce4af5b87767fed" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 39515 } } peers { permanent_uuid: "55c0ce5da3694b759b5b3113647c3eaa" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 41719 } } }
I20260620 14:48:04.742708 7086 sys_catalog.cc:458] T 00000000000000000000000000000000 P 55c0ce5da3694b759b5b3113647c3eaa [sys.catalog]: This master's current role is: FOLLOWER
I20260620 14:48:04.743505 7079 sys_catalog.cc:564] T 00000000000000000000000000000000 P 55c0ce5da3694b759b5b3113647c3eaa [sys.catalog]: configured and running, proceeding with master startup.
W20260620 14:48:04.752468 7097 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 55c0ce5da3694b759b5b3113647c3eaa: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260620 14:48:04.752661 7097 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260620 14:48:04.755349 7012 master_runner.cc:428] Detected that this master 55c0ce5da3694b759b5b3113647c3eaa is joining an existing cluster
I20260620 14:48:04.755470 7012 master_runner.cc:432] Initiating AddMaster RPC to add 127.0.97.61:41719
WARNING: no policy specified for oryx/127.0.97.60@KRBTEST.COM; defaulting to no policy
I20260620 14:48:04.788882 7083 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:48:04.763443 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57906 (local address 127.0.97.62:39515)
0620 14:48:04.763677 (+ 234us) server_negotiation.cc:206] Beginning negotiation
0620 14:48:04.763683 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 14:48:04.768318 (+ 4635us) server_negotiation.cc:407] Connection header received
0620 14:48:04.768597 (+ 279us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:48:04.768604 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:48:04.768705 (+ 101us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:48:04.768780 (+ 75us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:48:04.770318 (+ 1538us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:48:04.771425 (+ 1107us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:48:04.774028 (+ 2603us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:48:04.774724 (+ 696us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:48:04.774833 (+ 109us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:48:04.780721 (+ 5888us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 14:48:04.780728 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 14:48:04.780734 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 14:48:04.780781 (+ 47us) server_negotiation.cc:922] calling sasl_server_start()
0620 14:48:04.785322 (+ 4541us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:48:04.786214 (+ 892us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:48:04.786222 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:48:04.786227 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:48:04.786407 (+ 180us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:48:04.786940 (+ 533us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:48:04.786949 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:48:04.786953 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:48:04.787327 (+ 374us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 14:48:04.787477 (+ 150us) server_negotiation.cc:1035] Waiting for connection context
0620 14:48:04.788184 (+ 707us) server_negotiation.cc:299] Negotiation successful
0620 14:48:04.788446 (+ 262us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":55}
I20260620 14:48:04.789805 7103 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:48:04.762616 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:39515 (local address 127.0.0.1:57906)
0620 14:48:04.768173 (+ 5557us) negotiation.cc:107] Waiting for socket to connect
0620 14:48:04.768200 (+ 27us) client_negotiation.cc:174] Beginning negotiation
0620 14:48:04.768444 (+ 244us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 14:48:04.769009 (+ 565us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 14:48:04.769014 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 14:48:04.769710 (+ 696us) client_negotiation.cc:189] Negotiated authn=SASL
0620 14:48:04.770118 (+ 408us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:48:04.770138 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:48:04.771641 (+ 1503us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:48:04.771646 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:48:04.773866 (+ 2220us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:48:04.773880 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:48:04.775139 (+ 1259us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:48:04.775152 (+ 13us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:48:04.775397 (+ 245us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:48:04.778262 (+ 2865us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 14:48:04.778288 (+ 26us) client_negotiation.cc:656] calling sasl_client_start()
0620 14:48:04.779479 (+ 1191us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 14:48:04.785497 (+ 6018us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 14:48:04.785503 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 14:48:04.785508 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0620 14:48:04.786029 (+ 521us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 14:48:04.786574 (+ 545us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 14:48:04.786581 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 14:48:04.786586 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0620 14:48:04.786788 (+ 202us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 14:48:04.787576 (+ 788us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 14:48:04.787581 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 14:48:04.787875 (+ 294us) client_negotiation.cc:769] Sending connection context
0620 14:48:04.788968 (+ 1093us) client_negotiation.cc:240] Negotiation successful
0620 14:48:04.789331 (+ 363us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":5377,"mutex_wait_us":4,"thread_start_us":5218,"threads_started":1}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.0.97.60@KRBTEST.COM" created.
I20260620 14:48:04.795380 7082 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:48:04.763195 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48036 (local address 127.0.97.61:41719)
0620 14:48:04.763717 (+ 522us) server_negotiation.cc:206] Beginning negotiation
0620 14:48:04.763726 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0620 14:48:04.773233 (+ 9507us) server_negotiation.cc:407] Connection header received
0620 14:48:04.773512 (+ 279us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:48:04.773517 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:48:04.773612 (+ 95us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:48:04.773708 (+ 96us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:48:04.775814 (+ 2106us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:48:04.776869 (+ 1055us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:48:04.781542 (+ 4673us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:48:04.782325 (+ 783us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:48:04.782423 (+ 98us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:48:04.788397 (+ 5974us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 14:48:04.788406 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 14:48:04.788413 (+ 7us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 14:48:04.788455 (+ 42us) server_negotiation.cc:922] calling sasl_server_start()
0620 14:48:04.792313 (+ 3858us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:48:04.793200 (+ 887us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:48:04.793206 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:48:04.793209 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:48:04.793366 (+ 157us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:48:04.793771 (+ 405us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:48:04.793775 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:48:04.793778 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:48:04.794152 (+ 374us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 14:48:04.794287 (+ 135us) server_negotiation.cc:1035] Waiting for connection context
0620 14:48:04.794745 (+ 458us) server_negotiation.cc:299] Negotiation successful
0620 14:48:04.794983 (+ 238us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":331}
I20260620 14:48:04.796487 7104 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:48:04.762820 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.61:41719 (local address 127.0.0.1:48036)
0620 14:48:04.773105 (+ 10285us) negotiation.cc:107] Waiting for socket to connect
0620 14:48:04.773137 (+ 32us) client_negotiation.cc:174] Beginning negotiation
0620 14:48:04.773345 (+ 208us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 14:48:04.773971 (+ 626us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 14:48:04.773977 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 14:48:04.774882 (+ 905us) client_negotiation.cc:189] Negotiated authn=SASL
0620 14:48:04.775667 (+ 785us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:48:04.775680 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:48:04.777141 (+ 1461us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:48:04.777147 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:48:04.780470 (+ 3323us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:48:04.780487 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:48:04.783731 (+ 3244us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:48:04.783740 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:48:04.786345 (+ 2605us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:48:04.787023 (+ 678us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 14:48:04.787046 (+ 23us) client_negotiation.cc:656] calling sasl_client_start()
0620 14:48:04.788172 (+ 1126us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 14:48:04.792472 (+ 4300us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 14:48:04.792478 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 14:48:04.792481 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0620 14:48:04.793072 (+ 591us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 14:48:04.793499 (+ 427us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 14:48:04.793504 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 14:48:04.793507 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0620 14:48:04.793665 (+ 158us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 14:48:04.794353 (+ 688us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 14:48:04.794358 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 14:48:04.794492 (+ 134us) client_negotiation.cc:769] Sending connection context
0620 14:48:04.795655 (+ 1163us) client_negotiation.cc:240] Negotiation successful
0620 14:48:04.795955 (+ 300us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":10072,"mutex_wait_us":303,"thread_start_us":9795,"threads_started":1}
Jun 20 14:48:04 dist-test-slave-q9j6 krb5kdc[6505](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781966882, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.97.62@KRBTEST.COM for oryx/127.0.97.61@KRBTEST.COM
I20260620 14:48:04.811439 7086 raft_consensus.cc:482] T 00000000000000000000000000000000 P 55c0ce5da3694b759b5b3113647c3eaa [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260620 14:48:04.811770 7086 raft_consensus.cc:504] T 00000000000000000000000000000000 P 55c0ce5da3694b759b5b3113647c3eaa [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d663f8b7f54d4fe68ce4af5b87767fed" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 39515 } } peers { permanent_uuid: "55c0ce5da3694b759b5b3113647c3eaa" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 41719 } }
I20260620 14:48:04.813395 7086 leader_election.cc:288] T 00000000000000000000000000000000 P 55c0ce5da3694b759b5b3113647c3eaa [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers d663f8b7f54d4fe68ce4af5b87767fed (127.0.97.62:39515)
I20260620 14:48:04.814283 6560 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "55c0ce5da3694b759b5b3113647c3eaa" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d663f8b7f54d4fe68ce4af5b87767fed" is_pre_election: true
I20260620 14:48:04.815480 7029 leader_election.cc:302] T 00000000000000000000000000000000 P 55c0ce5da3694b759b5b3113647c3eaa [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 2 responses out of 2 voters: 1 yes votes; 1 no votes. yes voters: 55c0ce5da3694b759b5b3113647c3eaa; no voters: d663f8b7f54d4fe68ce4af5b87767fed
I20260620 14:48:04.816251 7086 raft_consensus.cc:2726] T 00000000000000000000000000000000 P 55c0ce5da3694b759b5b3113647c3eaa [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260620 14:48:04.816668 7082 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:48:04.797701 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48040 (local address 127.0.97.61:41719)
0620 14:48:04.797881 (+ 180us) server_negotiation.cc:206] Beginning negotiation
0620 14:48:04.797889 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0620 14:48:04.800342 (+ 2453us) server_negotiation.cc:407] Connection header received
0620 14:48:04.800752 (+ 410us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:48:04.800757 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:48:04.800874 (+ 117us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:48:04.800998 (+ 124us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:48:04.802458 (+ 1460us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:48:04.803220 (+ 762us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:48:04.804613 (+ 1393us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:48:04.805075 (+ 462us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:48:04.805251 (+ 176us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:48:04.808610 (+ 3359us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 14:48:04.808615 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 14:48:04.808618 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 14:48:04.808661 (+ 43us) server_negotiation.cc:922] calling sasl_server_start()
0620 14:48:04.812131 (+ 3470us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:48:04.813099 (+ 968us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:48:04.813105 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:48:04.813109 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:48:04.813282 (+ 173us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:48:04.813862 (+ 580us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:48:04.813867 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:48:04.813870 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:48:04.814221 (+ 351us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 14:48:04.814364 (+ 143us) server_negotiation.cc:1035] Waiting for connection context
0620 14:48:04.815890 (+ 1526us) server_negotiation.cc:299] Negotiation successful
0620 14:48:04.816142 (+ 252us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":57}
I20260620 14:48:04.817080 7106 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:48:04.797326 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.61:41719 (local address 127.0.0.1:48040)
0620 14:48:04.800199 (+ 2873us) negotiation.cc:107] Waiting for socket to connect
0620 14:48:04.800248 (+ 49us) client_negotiation.cc:174] Beginning negotiation
0620 14:48:04.800483 (+ 235us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 14:48:04.801071 (+ 588us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 14:48:04.801082 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 14:48:04.801983 (+ 901us) client_negotiation.cc:189] Negotiated authn=SASL
0620 14:48:04.802284 (+ 301us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:48:04.802298 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:48:04.803362 (+ 1064us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:48:04.803371 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:48:04.804460 (+ 1089us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:48:04.804472 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:48:04.805293 (+ 821us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:48:04.805299 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:48:04.805493 (+ 194us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:48:04.806176 (+ 683us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 14:48:04.806194 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0620 14:48:04.808397 (+ 2203us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 14:48:04.812328 (+ 3931us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 14:48:04.812343 (+ 15us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 14:48:04.812350 (+ 7us) client_negotiation.cc:760] calling sasl_client_step()
0620 14:48:04.812943 (+ 593us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 14:48:04.813434 (+ 491us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 14:48:04.813438 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 14:48:04.813442 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0620 14:48:04.813726 (+ 284us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 14:48:04.814443 (+ 717us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 14:48:04.814451 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 14:48:04.814833 (+ 382us) client_negotiation.cc:769] Sending connection context
0620 14:48:04.815018 (+ 185us) client_negotiation.cc:240] Negotiation successful
0620 14:48:04.815376 (+ 358us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":2624,"thread_start_us":2470,"threads_started":1}
I20260620 14:48:04.818890 6549 catalog_manager.cc:6979] Initiating ChangeConfig request to add master 127.0.97.61:41719: tablet_id: "00000000000000000000000000000000"
type: ADD_PEER
server {
permanent_uuid: "55c0ce5da3694b759b5b3113647c3eaa"
member_type: NON_VOTER
last_known_addr {
host: "127.0.97.61"
port: 41719
}
attrs {
promote: true
}
}
dest_uuid: "d663f8b7f54d4fe68ce4af5b87767fed"
cas_config_opid_index: -1
I20260620 14:48:04.820122 6549 consensus_queue.cc:237] T 00000000000000000000000000000000 P d663f8b7f54d4fe68ce4af5b87767fed [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "d663f8b7f54d4fe68ce4af5b87767fed" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 39515 } } peers { permanent_uuid: "55c0ce5da3694b759b5b3113647c3eaa" member_type: NON_VOTER last_known_addr { host: "127.0.97.61" port: 41719 } attrs { promote: true } }
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.0.97.60 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskUci3DU/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781966829091300-388-0/krb5kdc/oryx_127.0.97.60.keytab.
Entry for principal oryx/127.0.97.60 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskUci3DU/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781966829091300-388-0/krb5kdc/oryx_127.0.97.60.keytab.
I20260620 14:48:04.823925 7110 raft_consensus.cc:2932] T 00000000000000000000000000000000 P d663f8b7f54d4fe68ce4af5b87767fed [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index -1 to 5, NON_VOTER 55c0ce5da3694b759b5b3113647c3eaa (127.0.97.61) added. New config: { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "d663f8b7f54d4fe68ce4af5b87767fed" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 39515 } } peers { permanent_uuid: "55c0ce5da3694b759b5b3113647c3eaa" member_type: NON_VOTER last_known_addr { host: "127.0.97.61" port: 41719 } attrs { promote: true } } }
I20260620 14:48:04.824978 7054 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 55c0ce5da3694b759b5b3113647c3eaa [term 0 FOLLOWER]: Advancing to term 1
I20260620 14:48:04.825755 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskUci3DU/build/asan/bin/kudu
/tmp/dist-test-taskUci3DU/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskUci3DU/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781966829091300-388-0/minicluster-data/master-2/wal
--fs_data_dirs=/tmp/dist-test-taskUci3DU/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781966829091300-388-0/minicluster-data/master-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskUci3DU/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781966829091300-388-0/minicluster-data/master-2/logs
--server_dump_info_path=/tmp/dist-test-taskUci3DU/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781966829091300-388-0/minicluster-data/master-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.60:37899
--webserver_interface=127.0.97.60
--webserver_port=0
--keytab_file=/tmp/dist-test-taskUci3DU/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781966829091300-388-0/krb5kdc/oryx_127.0.97.60.keytab
--principal=oryx/127.0.97.60
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:33997
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.97.62:39515,127.0.97.61:41719,127.0.97.60:37899
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskUci3DU/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781966829091300-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskUci3DU/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781966829091300-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskUci3DU/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781966829091300-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260620 14:48:04.828243 7110 catalog_manager.cc:6983] Successfully completed master ChangeConfig request to add master 127.0.97.61:41719
I20260620 14:48:04.828819 7054 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 55c0ce5da3694b759b5b3113647c3eaa [term 1 FOLLOWER]: Refusing update from remote peer d663f8b7f54d4fe68ce4af5b87767fed: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20260620 14:48:04.829057 7012 master.cc:537] Master@127.0.97.61:41719 shutting down...
I20260620 14:48:04.829653 7110 sys_catalog.cc:455] T 00000000000000000000000000000000 P d663f8b7f54d4fe68ce4af5b87767fed [sys.catalog]: SysCatalogTable state changed. Reason: Config change replication complete. Latest consensus state: current_term: 1 leader_uuid: "d663f8b7f54d4fe68ce4af5b87767fed" committed_config { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "d663f8b7f54d4fe68ce4af5b87767fed" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 39515 } } peers { permanent_uuid: "55c0ce5da3694b759b5b3113647c3eaa" member_type: NON_VOTER last_known_addr { host: "127.0.97.61" port: 41719 } attrs { promote: true } } }
I20260620 14:48:04.829890 7110 sys_catalog.cc:458] T 00000000000000000000000000000000 P d663f8b7f54d4fe68ce4af5b87767fed [sys.catalog]: This master's current role is: LEADER
I20260620 14:48:04.830950 7109 consensus_queue.cc:1035] T 00000000000000000000000000000000 P d663f8b7f54d4fe68ce4af5b87767fed [LEADER]: Connected to new peer: Peer: permanent_uuid: "55c0ce5da3694b759b5b3113647c3eaa" member_type: NON_VOTER last_known_addr { host: "127.0.97.61" port: 41719 } attrs { promote: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 0, Time since last communication: 0.000s
W20260620 14:48:04.834334 6537 proxy.cc:239] Call had error, refreshing address and retrying: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on Master
W20260620 14:48:04.838079 6537 consensus_peers.cc:487] T 00000000000000000000000000000000 P d663f8b7f54d4fe68ce4af5b87767fed -> Peer 55c0ce5da3694b759b5b3113647c3eaa (127.0.97.61:41719): Couldn't send request to peer 55c0ce5da3694b759b5b3113647c3eaa. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on Master. This is attempt 1: this message will repeat every 5th retry.
I20260620 14:48:04.841361 7012 raft_consensus.cc:2229] T 00000000000000000000000000000000 P 55c0ce5da3694b759b5b3113647c3eaa [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 14:48:04.841773 7012 raft_consensus.cc:2258] T 00000000000000000000000000000000 P 55c0ce5da3694b759b5b3113647c3eaa [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 14:48:04.841974 7012 tablet_replica.cc:330] T 00000000000000000000000000000000 P 55c0ce5da3694b759b5b3113647c3eaa: stopping tablet replica
I20260620 14:48:04.848402 7012 master.cc:552] Master@127.0.97.61:41719 shutdown complete.
I20260620 14:48:04.848510 7012 master_runner.cc:305] Clearing existing system tablet
I20260620 14:48:04.848886 7012 fs_manager.cc:359] Metadata directory not provided
I20260620 14:48:04.848995 7012 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 14:48:04.851194 7012 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.003s sys 0.000s
I20260620 14:48:04.853065 7115 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 14:48:04.853456 7012 fs_report.cc:383] FS layout report
--------------------
wal directory:
metadata directory:
1 data directories: /tmp/dist-test-taskUci3DU/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781966829091300-388-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 14:48:04.853745 7012 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.000s
I20260620 14:48:04.853835 7012 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskUci3DU/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781966829091300-388-0/minicluster-data/master-1/data,/tmp/dist-test-taskUci3DU/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781966829091300-388-0/minicluster-data/master-1/wal
uuid: "55c0ce5da3694b759b5b3113647c3eaa"
format_stamp: "Formatted at 2026-06-20 14:48:04 on dist-test-slave-q9j6"
server_key: "3d85e1244253c50a1bcdb8d6f9abcd11"
server_key_iv: "2e529241fa53477dbdceb9fa62309930"
server_key_version: "encryptionkey@0"
I20260620 14:48:04.858445 7012 ts_tablet_manager.cc:1905] T 00000000000000000000000000000000 P 55c0ce5da3694b759b5b3113647c3eaa: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 14:48:04.864308 7012 ts_tablet_manager.cc:1918] T 00000000000000000000000000000000 P 55c0ce5da3694b759b5b3113647c3eaa: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId unknown
I20260620 14:48:04.864500 7012 log.cc:1196] T 00000000000000000000000000000000 P 55c0ce5da3694b759b5b3113647c3eaa: Deleting WAL directory at /tmp/dist-test-taskUci3DU/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781966829091300-388-0/minicluster-data/master-1/wal/wals/00000000000000000000000000000000
I20260620 14:48:04.864907 7012 ts_tablet_manager.cc:1939] T 00000000000000000000000000000000 P 55c0ce5da3694b759b5b3113647c3eaa: Deleting consensus metadata
I20260620 14:48:04.865170 7012 master_runner.cc:315] Copying system tablet from 127.0.97.62:39515
I20260620 14:48:04.873210 7012 tablet_copy_client.cc:316] T 00000000000000000000000000000000 P 55c0ce5da3694b759b5b3113647c3eaa: tablet copy: Beginning tablet copy session from remote peer at address 127.0.97.62:39515
I20260620 14:48:04.886433 7083 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:48:04.874042 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57918 (local address 127.0.97.62:39515)
0620 14:48:04.874269 (+ 227us) server_negotiation.cc:206] Beginning negotiation
0620 14:48:04.874279 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0620 14:48:04.874769 (+ 490us) server_negotiation.cc:407] Connection header received
0620 14:48:04.874986 (+ 217us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:48:04.874992 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:48:04.875091 (+ 99us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:48:04.875204 (+ 113us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:48:04.876087 (+ 883us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:48:04.876863 (+ 776us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:48:04.878534 (+ 1671us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:48:04.879132 (+ 598us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:48:04.879269 (+ 137us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:48:04.881145 (+ 1876us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 14:48:04.881151 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 14:48:04.881156 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 14:48:04.881193 (+ 37us) server_negotiation.cc:922] calling sasl_server_start()
0620 14:48:04.883864 (+ 2671us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:48:04.884478 (+ 614us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:48:04.884485 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:48:04.884488 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:48:04.884616 (+ 128us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:48:04.884973 (+ 357us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:48:04.884980 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:48:04.884985 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:48:04.885255 (+ 270us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 14:48:04.885416 (+ 161us) server_negotiation.cc:1035] Waiting for connection context
0620 14:48:04.885806 (+ 390us) server_negotiation.cc:299] Negotiation successful
0620 14:48:04.886021 (+ 215us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":81}
I20260620 14:48:04.886466 7121 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:48:04.874030 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:39515 (local address 127.0.0.1:57918)
0620 14:48:04.874646 (+ 616us) negotiation.cc:107] Waiting for socket to connect
0620 14:48:04.874662 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0620 14:48:04.874849 (+ 187us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 14:48:04.875231 (+ 382us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 14:48:04.875235 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 14:48:04.875711 (+ 476us) client_negotiation.cc:189] Negotiated authn=SASL
0620 14:48:04.875929 (+ 218us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:48:04.875938 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:48:04.876983 (+ 1045us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:48:04.876989 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:48:04.878359 (+ 1370us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:48:04.878376 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:48:04.879287 (+ 911us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:48:04.879294 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:48:04.879463 (+ 169us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:48:04.880225 (+ 762us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 14:48:04.880242 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0620 14:48:04.880949 (+ 707us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 14:48:04.883996 (+ 3047us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 14:48:04.884003 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 14:48:04.884007 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0620 14:48:04.884357 (+ 350us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 14:48:04.884726 (+ 369us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 14:48:04.884731 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 14:48:04.884734 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0620 14:48:04.884868 (+ 134us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 14:48:04.885379 (+ 511us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 14:48:04.885384 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 14:48:04.885557 (+ 173us) client_negotiation.cc:769] Sending connection context
0620 14:48:04.885706 (+ 149us) client_negotiation.cc:240] Negotiation successful
0620 14:48:04.886028 (+ 322us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":460,"thread_start_us":369,"threads_started":1}
I20260620 14:48:04.887574 6570 tablet_copy_service.cc:140] P d663f8b7f54d4fe68ce4af5b87767fed: Received BeginTabletCopySession request for tablet 00000000000000000000000000000000 from peer 55c0ce5da3694b759b5b3113647c3eaa ({username='oryx', principal='oryx/127.0.97.61@KRBTEST.COM'} at 127.0.0.1:57918)
I20260620 14:48:04.887899 6570 tablet_copy_service.cc:161] P d663f8b7f54d4fe68ce4af5b87767fed: Beginning new tablet copy session on tablet 00000000000000000000000000000000 from peer 55c0ce5da3694b759b5b3113647c3eaa at {username='oryx', principal='oryx/127.0.97.61@KRBTEST.COM'} at 127.0.0.1:57918: session id = 55c0ce5da3694b759b5b3113647c3eaa-00000000000000000000000000000000
I20260620 14:48:04.891021 6570 tablet_copy_source_session.cc:205] T 00000000000000000000000000000000 P d663f8b7f54d4fe68ce4af5b87767fed: Tablet Copy: opened 0 blocks and 1 log segments
I20260620 14:48:04.893143 7012 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total
I20260620 14:48:04.897810 7012 tablet_copy_client.cc:791] T 00000000000000000000000000000000 P 55c0ce5da3694b759b5b3113647c3eaa: tablet copy: Starting download of 0 data blocks...
I20260620 14:48:04.898144 7012 tablet_copy_client.cc:655] T 00000000000000000000000000000000 P 55c0ce5da3694b759b5b3113647c3eaa: tablet copy: Starting download of 1 WAL segments...
I20260620 14:48:04.900789 7012 tablet_copy_client.cc:531] T 00000000000000000000000000000000 P 55c0ce5da3694b759b5b3113647c3eaa: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20260620 14:48:04.904132 6570 tablet_copy_service.cc:342] P d663f8b7f54d4fe68ce4af5b87767fed: Request end of tablet copy session 55c0ce5da3694b759b5b3113647c3eaa-00000000000000000000000000000000 received from {username='oryx', principal='oryx/127.0.97.61@KRBTEST.COM'} at 127.0.0.1:57918
I20260620 14:48:04.904320 6570 tablet_copy_service.cc:433] P d663f8b7f54d4fe68ce4af5b87767fed: ending tablet copy session 55c0ce5da3694b759b5b3113647c3eaa-00000000000000000000000000000000 on tablet 00000000000000000000000000000000 with peer 55c0ce5da3694b759b5b3113647c3eaa
I20260620 14:48:04.914700 7012 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 14:48:04.917496 7122 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 14:48:04.921178 7123 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 14:48:04.931937 7125 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 14:48:04.933338 7012 server_base.cc:1034] running on GCE node
I20260620 14:48:04.933717 7012 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 14:48:04.934441 7012 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 14:48:04.935693 7012 hybrid_clock.cc:648] HybridClock initialized: now 1781966884935615 us; error 49 us; skew 500 ppm
I20260620 14:48:04.935968 7012 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 14:48:04 dist-test-slave-q9j6 krb5kdc[6505](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781966884, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.97.61@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 14:48:04.937875 7012 init.cc:377] Logged in from keytab as oryx/127.0.97.61@KRBTEST.COM (short username oryx)
I20260620 14:48:04.939218 7012 webserver.cc:460] Webserver started at http://127.0.97.61:35103/ using document root <none> and password file <none>
I20260620 14:48:04.939541 7012 fs_manager.cc:359] Metadata directory not provided
I20260620 14:48:04.939706 7012 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 14:48:04.942205 7012 fs_manager.cc:709] Time spent opening directory manager: real 0.002s user 0.000s sys 0.002s
I20260620 14:48:04.943949 7132 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 14:48:04.944428 7012 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.001s
I20260620 14:48:04.944579 7012 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskUci3DU/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781966829091300-388-0/minicluster-data/master-1/data,/tmp/dist-test-taskUci3DU/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781966829091300-388-0/minicluster-data/master-1/wal
uuid: "55c0ce5da3694b759b5b3113647c3eaa"
format_stamp: "Formatted at 2026-06-20 14:48:04 on dist-test-slave-q9j6"
server_key: "3d85e1244253c50a1bcdb8d6f9abcd11"
server_key_iv: "2e529241fa53477dbdceb9fa62309930"
server_key_version: "encryptionkey@0"
I20260620 14:48:04.944787 7012 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskUci3DU/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781966829091300-388-0/minicluster-data/master-1/wal
metadata directory: /tmp/dist-test-taskUci3DU/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781966829091300-388-0/minicluster-data/master-1/wal
1 data directories: /tmp/dist-test-taskUci3DU/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781966829091300-388-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 14:48:04.958227 7012 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 14:48:04.962123 7012 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 14:48:04.994352 7012 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.61:41719
I20260620 14:48:04.994375 7183 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.61:41719 every 8 connection(s)
I20260620 14:48:04.995265 7012 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskUci3DU/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781966829091300-388-0/minicluster-data/master-1/data/info.pb
I20260620 14:48:05.002315 7184 sys_catalog.cc:263] Verifying existing consensus state
I20260620 14:48:05.005101 7184 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 55c0ce5da3694b759b5b3113647c3eaa: Bootstrap starting.
I20260620 14:48:05.018548 7184 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 55c0ce5da3694b759b5b3113647c3eaa: Bootstrap replayed 1/1 log segments. Stats: ops{read=5 overwritten=0 applied=5 ignored=0} inserts{seen=3 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260620 14:48:05.019088 7184 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 55c0ce5da3694b759b5b3113647c3eaa: Bootstrap complete.
I20260620 14:48:05.020084 7184 raft_consensus.cc:348] T 00000000000000000000000000000000 P 55c0ce5da3694b759b5b3113647c3eaa [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "d663f8b7f54d4fe68ce4af5b87767fed" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 39515 } } peers { permanent_uuid: "55c0ce5da3694b759b5b3113647c3eaa" member_type: NON_VOTER last_known_addr { host: "127.0.97.61" port: 41719 } attrs { promote: true } }
I20260620 14:48:05.020337 7184 raft_consensus.cc:729] T 00000000000000000000000000000000 P 55c0ce5da3694b759b5b3113647c3eaa [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 55c0ce5da3694b759b5b3113647c3eaa, State: Initialized, Role: LEARNER
I20260620 14:48:05.020581 7184 consensus_queue.cc:260] T 00000000000000000000000000000000 P 55c0ce5da3694b759b5b3113647c3eaa [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 5, Last appended: 1.5, Last appended by leader: 5, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "d663f8b7f54d4fe68ce4af5b87767fed" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 39515 } } peers { permanent_uuid: "55c0ce5da3694b759b5b3113647c3eaa" member_type: NON_VOTER last_known_addr { host: "127.0.97.61" port: 41719 } attrs { promote: true } }
I20260620 14:48:05.021397 7187 sys_catalog.cc:455] T 00000000000000000000000000000000 P 55c0ce5da3694b759b5b3113647c3eaa [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 committed_config { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "d663f8b7f54d4fe68ce4af5b87767fed" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 39515 } } peers { permanent_uuid: "55c0ce5da3694b759b5b3113647c3eaa" member_type: NON_VOTER last_known_addr { host: "127.0.97.61" port: 41719 } attrs { promote: true } } }
I20260620 14:48:05.021759 7184 sys_catalog.cc:564] T 00000000000000000000000000000000 P 55c0ce5da3694b759b5b3113647c3eaa [sys.catalog]: configured and running, proceeding with master startup.
I20260620 14:48:05.021859 7187 sys_catalog.cc:458] T 00000000000000000000000000000000 P 55c0ce5da3694b759b5b3113647c3eaa [sys.catalog]: This master's current role is: LEARNER
I20260620 14:48:05.034130 7198 catalog_manager.cc:1259] Loaded cluster ID: a48bc503bb2347ef82ed8e77d4e21c70
I20260620 14:48:05.034273 7198 catalog_manager.cc:1552] T 00000000000000000000000000000000 P 55c0ce5da3694b759b5b3113647c3eaa: loading cluster ID for follower catalog manager: success
I20260620 14:48:05.036798 7198 catalog_manager.cc:1574] T 00000000000000000000000000000000 P 55c0ce5da3694b759b5b3113647c3eaa: acquiring CA information for follower catalog manager: success
I20260620 14:48:05.039330 7198 catalog_manager.cc:1602] T 00000000000000000000000000000000 P 55c0ce5da3694b759b5b3113647c3eaa: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260620 14:48:05.165963 7007 heartbeater.cc:503] Master 127.0.97.62:39515 was elected leader, sending a full tablet report...
W20260620 14:48:05.278151 7112 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 14:48:05.278548 7112 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 14:48:05.278713 7112 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 14:48:05.286489 7112 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 14:48:05.286656 7112 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 14:48:05.286767 7112 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 14:48:05.286868 7112 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 14:48:05.286942 7112 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 14:48:05.297561 7112 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:33997
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskUci3DU/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781966829091300-388-0/minicluster-data/master-2/data
--fs_wal_dir=/tmp/dist-test-taskUci3DU/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781966829091300-388-0/minicluster-data/master-2/wal
--ipki_ca_key_size=768
--master_addresses=127.0.97.62:39515,127.0.97.61:41719,127.0.97.60:37899
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskUci3DU/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781966829091300-388-0/krb5kdc/oryx_127.0.97.60.keytab
--principal=oryx/127.0.97.60
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.60:37899
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskUci3DU/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781966829091300-388-0/minicluster-data/master-2/data/info.pb
--webserver_interface=127.0.97.60
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskUci3DU/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781966829091300-388-0/minicluster-data/master-2/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 13:41:20 UTC on ccycloud.quasar-vdagoi.root.comops.site
build id 4457462
ASAN enabled
I20260620 14:48:05.298976 7112 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 14:48:05.300446 7112 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 14:48:05.311393 7203 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 14:48:05.312234 7204 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 14:48:05.313529 7206 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 14:48:05.314077 7112 server_base.cc:1034] running on GCE node
I20260620 14:48:05.314927 7112 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 14:48:05.316435 7112 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 14:48:05.317674 7112 hybrid_clock.cc:648] HybridClock initialized: now 1781966885317635 us; error 59 us; skew 500 ppm
I20260620 14:48:05.318041 7112 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 14:48:05 dist-test-slave-q9j6 krb5kdc[6505](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781966885, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.97.60@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 14:48:05.321146 7112 init.cc:377] Logged in from keytab as oryx/127.0.97.60@KRBTEST.COM (short username oryx)
I20260620 14:48:05.322805 7112 webserver.cc:460] Webserver started at http://127.0.97.60:42213/ using document root <none> and password file <none>
I20260620 14:48:05.323527 7112 fs_manager.cc:359] Metadata directory not provided
I20260620 14:48:05.323696 7112 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 14:48:05.323999 7112 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 14:48:05.326885 7112 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskUci3DU/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781966829091300-388-0/minicluster-data/master-2/data/instance:
uuid: "e4dec65330ba4aefa643b5c1fcbc31ae"
format_stamp: "Formatted at 2026-06-20 14:48:05 on dist-test-slave-q9j6"
server_key: "b2b7f353c82b330cb4f21c9397e5ee9a"
server_key_iv: "f3daaba5e81f9394bbaba79b961423ba"
server_key_version: "encryptionkey@0"
I20260620 14:48:05.327729 7112 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskUci3DU/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781966829091300-388-0/minicluster-data/master-2/wal/instance:
uuid: "e4dec65330ba4aefa643b5c1fcbc31ae"
format_stamp: "Formatted at 2026-06-20 14:48:05 on dist-test-slave-q9j6"
server_key: "b2b7f353c82b330cb4f21c9397e5ee9a"
server_key_iv: "f3daaba5e81f9394bbaba79b961423ba"
server_key_version: "encryptionkey@0"
I20260620 14:48:05.332926 7112 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.004s sys 0.000s
I20260620 14:48:05.337090 7213 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 14:48:05.338629 7112 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260620 14:48:05.338838 7112 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskUci3DU/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781966829091300-388-0/minicluster-data/master-2/data,/tmp/dist-test-taskUci3DU/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781966829091300-388-0/minicluster-data/master-2/wal
uuid: "e4dec65330ba4aefa643b5c1fcbc31ae"
format_stamp: "Formatted at 2026-06-20 14:48:05 on dist-test-slave-q9j6"
server_key: "b2b7f353c82b330cb4f21c9397e5ee9a"
server_key_iv: "f3daaba5e81f9394bbaba79b961423ba"
server_key_version: "encryptionkey@0"
I20260620 14:48:05.339057 7112 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskUci3DU/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781966829091300-388-0/minicluster-data/master-2/wal
metadata directory: /tmp/dist-test-taskUci3DU/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781966829091300-388-0/minicluster-data/master-2/wal
1 data directories: /tmp/dist-test-taskUci3DU/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781966829091300-388-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 14:48:05.356827 7112 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 14:48:05.361236 7112 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 14:48:05.361537 7112 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 14:48:05.382275 7112 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.60:37899
I20260620 14:48:05.382298 7264 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.60:37899 every 8 connection(s)
I20260620 14:48:05.383817 7112 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskUci3DU/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781966829091300-388-0/minicluster-data/master-2/data/info.pb
I20260620 14:48:05.387283 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskUci3DU/build/asan/bin/kudu as pid 7112
I20260620 14:48:05.388185 7265 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 14:48:05.400763 7265 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.97.62" port: 39515 } has no permanent_uuid. Determining permanent_uuid...
I20260620 14:48:05.412230 7270 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:48:05.389046 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48054 (local address 127.0.97.61:41719)
0620 14:48:05.404107 (+ 15061us) server_negotiation.cc:206] Beginning negotiation
0620 14:48:05.404113 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 14:48:05.404137 (+ 24us) server_negotiation.cc:407] Connection header received
0620 14:48:05.404209 (+ 72us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:48:05.404214 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:48:05.404319 (+ 105us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:48:05.404458 (+ 139us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 14:48:05.405219 (+ 761us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:48:05.406835 (+ 1616us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:48:05.409331 (+ 2496us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:48:05.410617 (+ 1286us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:48:05.410749 (+ 132us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:48:05.411024 (+ 275us) server_negotiation.cc:1035] Waiting for connection context
0620 14:48:05.411692 (+ 668us) server_negotiation.cc:299] Negotiation successful
0620 14:48:05.411850 (+ 158us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":14916,"thread_start_us":14786,"threads_started":1}
I20260620 14:48:05.413246 7269 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:48:05.388644 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.61:41719 (local address 127.0.0.1:48054)
0620 14:48:05.403745 (+ 15101us) negotiation.cc:107] Waiting for socket to connect
0620 14:48:05.403780 (+ 35us) client_negotiation.cc:174] Beginning negotiation
0620 14:48:05.403962 (+ 182us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 14:48:05.404761 (+ 799us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 14:48:05.404768 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 14:48:05.404804 (+ 36us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 14:48:05.405024 (+ 220us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:48:05.405038 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:48:05.407042 (+ 2004us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:48:05.407048 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:48:05.409148 (+ 2100us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:48:05.409164 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:48:05.411215 (+ 2051us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:48:05.411222 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:48:05.411427 (+ 205us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:48:05.411497 (+ 70us) client_negotiation.cc:769] Sending connection context
0620 14:48:05.412761 (+ 1264us) client_negotiation.cc:240] Negotiation successful
0620 14:48:05.412926 (+ 165us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":14901,"thread_start_us":15434,"threads_started":1}
I20260620 14:48:05.415302 7159 raft_consensus.cc:1203] T 00000000000000000000000000000000 P 55c0ce5da3694b759b5b3113647c3eaa [term 1 LEARNER]: Deduplicated request from leader. Original: 0.0->[1.1-1.5] Dedup: 1.5->[]
I20260620 14:48:05.416242 7187 sys_catalog.cc:455] T 00000000000000000000000000000000 P 55c0ce5da3694b759b5b3113647c3eaa [sys.catalog]: SysCatalogTable state changed. Reason: New leader d663f8b7f54d4fe68ce4af5b87767fed. Latest consensus state: current_term: 1 leader_uuid: "d663f8b7f54d4fe68ce4af5b87767fed" committed_config { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "d663f8b7f54d4fe68ce4af5b87767fed" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 39515 } } peers { permanent_uuid: "55c0ce5da3694b759b5b3113647c3eaa" member_type: NON_VOTER last_known_addr { host: "127.0.97.61" port: 41719 } attrs { promote: true } } }
I20260620 14:48:05.416501 7187 sys_catalog.cc:458] T 00000000000000000000000000000000 P 55c0ce5da3694b759b5b3113647c3eaa [sys.catalog]: This master's current role is: LEARNER
Jun 20 14:48:05 dist-test-slave-q9j6 krb5kdc[6505](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781966882, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.0.97.60@KRBTEST.COM
Jun 20 14:48:05 dist-test-slave-q9j6 krb5kdc[6505](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781966885, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.97.60@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM
I20260620 14:48:05.430910 7271 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:48:05.389791 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51852 (local address 127.0.97.60:37899)
0620 14:48:05.401518 (+ 11727us) server_negotiation.cc:206] Beginning negotiation
0620 14:48:05.401543 (+ 25us) server_negotiation.cc:399] Waiting for connection header
0620 14:48:05.404856 (+ 3313us) server_negotiation.cc:407] Connection header received
0620 14:48:05.408152 (+ 3296us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:48:05.408187 (+ 35us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:48:05.408787 (+ 600us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:48:05.409244 (+ 457us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:48:05.411831 (+ 2587us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:48:05.413948 (+ 2117us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:48:05.415259 (+ 1311us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:48:05.416752 (+ 1493us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:48:05.417586 (+ 834us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:48:05.422346 (+ 4760us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 14:48:05.422355 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 14:48:05.422369 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 14:48:05.422412 (+ 43us) server_negotiation.cc:922] calling sasl_server_start()
0620 14:48:05.426494 (+ 4082us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:48:05.427307 (+ 813us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:48:05.427316 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:48:05.427327 (+ 11us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:48:05.427547 (+ 220us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:48:05.428009 (+ 462us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:48:05.428015 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:48:05.428019 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:48:05.428581 (+ 562us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 14:48:05.428902 (+ 321us) server_negotiation.cc:1035] Waiting for connection context
0620 14:48:05.429389 (+ 487us) server_negotiation.cc:299] Negotiation successful
0620 14:48:05.429818 (+ 429us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":11357,"thread_start_us":11682,"threads_started":1}
I20260620 14:48:05.432747 7273 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:48:05.405377 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57930 (local address 127.0.97.62:39515)
0620 14:48:05.415603 (+ 10226us) server_negotiation.cc:206] Beginning negotiation
0620 14:48:05.415609 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 14:48:05.416353 (+ 744us) server_negotiation.cc:407] Connection header received
0620 14:48:05.416680 (+ 327us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:48:05.416688 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:48:05.416768 (+ 80us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:48:05.416848 (+ 80us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:48:05.418517 (+ 1669us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:48:05.419924 (+ 1407us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:48:05.421412 (+ 1488us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:48:05.422085 (+ 673us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:48:05.422247 (+ 162us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:48:05.426797 (+ 4550us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 14:48:05.426802 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 14:48:05.426805 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 14:48:05.426841 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0620 14:48:05.429882 (+ 3041us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:48:05.430616 (+ 734us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:48:05.430622 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:48:05.430625 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:48:05.430762 (+ 137us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:48:05.431195 (+ 433us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:48:05.431202 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:48:05.431208 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:48:05.431562 (+ 354us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 14:48:05.431749 (+ 187us) server_negotiation.cc:1035] Waiting for connection context
0620 14:48:05.432238 (+ 489us) server_negotiation.cc:299] Negotiation successful
0620 14:48:05.432458 (+ 220us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":10094,"thread_start_us":11648,"threads_started":1}
I20260620 14:48:05.432766 7272 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:48:05.404927 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:39515 (local address 127.0.0.1:57930)
0620 14:48:05.416225 (+ 11298us) negotiation.cc:107] Waiting for socket to connect
0620 14:48:05.416272 (+ 47us) client_negotiation.cc:174] Beginning negotiation
0620 14:48:05.416502 (+ 230us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 14:48:05.417205 (+ 703us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 14:48:05.417213 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 14:48:05.418067 (+ 854us) client_negotiation.cc:189] Negotiated authn=SASL
0620 14:48:05.418324 (+ 257us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:48:05.418333 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:48:05.420074 (+ 1741us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:48:05.420078 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:48:05.421240 (+ 1162us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:48:05.421255 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:48:05.422355 (+ 1100us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:48:05.422363 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:48:05.422589 (+ 226us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:48:05.423358 (+ 769us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 14:48:05.423387 (+ 29us) client_negotiation.cc:656] calling sasl_client_start()
0620 14:48:05.426527 (+ 3140us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 14:48:05.430016 (+ 3489us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 14:48:05.430029 (+ 13us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 14:48:05.430036 (+ 7us) client_negotiation.cc:760] calling sasl_client_step()
0620 14:48:05.430486 (+ 450us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 14:48:05.430880 (+ 394us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 14:48:05.430885 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 14:48:05.430889 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0620 14:48:05.431035 (+ 146us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 14:48:05.431768 (+ 733us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 14:48:05.431773 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 14:48:05.431919 (+ 146us) client_negotiation.cc:769] Sending connection context
0620 14:48:05.432104 (+ 185us) client_negotiation.cc:240] Negotiation successful
0620 14:48:05.432346 (+ 242us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":11002,"thread_start_us":11303,"threads_started":1}
I20260620 14:48:05.434659 7265 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.97.61" port: 41719 } has no permanent_uuid. Determining permanent_uuid...
Jun 20 14:48:05 dist-test-slave-q9j6 krb5kdc[6505](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781966885, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.97.60@KRBTEST.COM for oryx/127.0.97.61@KRBTEST.COM
I20260620 14:48:05.448412 7270 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:48:05.435554 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48060 (local address 127.0.97.61:41719)
0620 14:48:05.435796 (+ 242us) server_negotiation.cc:206] Beginning negotiation
0620 14:48:05.435801 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 14:48:05.435817 (+ 16us) server_negotiation.cc:407] Connection header received
0620 14:48:05.435865 (+ 48us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:48:05.435868 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:48:05.435924 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:48:05.436006 (+ 82us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:48:05.436652 (+ 646us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:48:05.437355 (+ 703us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:48:05.438653 (+ 1298us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:48:05.439076 (+ 423us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:48:05.439178 (+ 102us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:48:05.442295 (+ 3117us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 14:48:05.442300 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 14:48:05.442304 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 14:48:05.442338 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0620 14:48:05.445329 (+ 2991us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:48:05.446150 (+ 821us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:48:05.446154 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:48:05.446156 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:48:05.446274 (+ 118us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:48:05.447111 (+ 837us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:48:05.447115 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:48:05.447117 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:48:05.447395 (+ 278us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 14:48:05.447504 (+ 109us) server_negotiation.cc:1035] Waiting for connection context
0620 14:48:05.447977 (+ 473us) server_negotiation.cc:299] Negotiation successful
0620 14:48:05.448164 (+ 187us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":128}
I20260620 14:48:05.448426 7272 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:48:05.435254 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.61:41719 (local address 127.0.0.1:48060)
0620 14:48:05.435446 (+ 192us) negotiation.cc:107] Waiting for socket to connect
0620 14:48:05.435470 (+ 24us) client_negotiation.cc:174] Beginning negotiation
0620 14:48:05.435549 (+ 79us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 14:48:05.436042 (+ 493us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 14:48:05.436045 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 14:48:05.436366 (+ 321us) client_negotiation.cc:189] Negotiated authn=SASL
0620 14:48:05.436504 (+ 138us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:48:05.436511 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:48:05.437510 (+ 999us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:48:05.437514 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:48:05.438484 (+ 970us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:48:05.438496 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:48:05.439219 (+ 723us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:48:05.439223 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:48:05.439348 (+ 125us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:48:05.440081 (+ 733us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 14:48:05.440100 (+ 19us) client_negotiation.cc:656] calling sasl_client_start()
0620 14:48:05.442088 (+ 1988us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 14:48:05.445526 (+ 3438us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 14:48:05.445530 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 14:48:05.445533 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0620 14:48:05.446035 (+ 502us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 14:48:05.446374 (+ 339us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 14:48:05.446377 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 14:48:05.446379 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 14:48:05.447012 (+ 633us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 14:48:05.447492 (+ 480us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 14:48:05.447495 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 14:48:05.447730 (+ 235us) client_negotiation.cc:769] Sending connection context
0620 14:48:05.447897 (+ 167us) client_negotiation.cc:240] Negotiation successful
0620 14:48:05.448214 (+ 317us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":58}
I20260620 14:48:05.449581 7265 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.97.60" port: 37899 } has no permanent_uuid. Determining permanent_uuid...
Jun 20 14:48:05 dist-test-slave-q9j6 krb5kdc[6505](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781966885, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.97.60@KRBTEST.COM for oryx/127.0.97.60@KRBTEST.COM
I20260620 14:48:05.461643 7272 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:48:05.450347 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.60:37899 (local address 127.0.0.1:51868)
0620 14:48:05.450547 (+ 200us) negotiation.cc:107] Waiting for socket to connect
0620 14:48:05.450563 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0620 14:48:05.450647 (+ 84us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 14:48:05.450880 (+ 233us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 14:48:05.450883 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 14:48:05.451194 (+ 311us) client_negotiation.cc:189] Negotiated authn=SASL
0620 14:48:05.451323 (+ 129us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:48:05.451330 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:48:05.452310 (+ 980us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:48:05.452314 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:48:05.453155 (+ 841us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:48:05.453166 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:48:05.453802 (+ 636us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:48:05.453806 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:48:05.453953 (+ 147us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:48:05.454572 (+ 619us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 14:48:05.454595 (+ 23us) client_negotiation.cc:656] calling sasl_client_start()
0620 14:48:05.456627 (+ 2032us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 14:48:05.459454 (+ 2827us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 14:48:05.459461 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 14:48:05.459469 (+ 8us) client_negotiation.cc:760] calling sasl_client_step()
0620 14:48:05.459902 (+ 433us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 14:48:05.460246 (+ 344us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 14:48:05.460249 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 14:48:05.460252 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0620 14:48:05.460393 (+ 141us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 14:48:05.460882 (+ 489us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 14:48:05.460885 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 14:48:05.461048 (+ 163us) client_negotiation.cc:769] Sending connection context
0620 14:48:05.461166 (+ 118us) client_negotiation.cc:240] Negotiation successful
0620 14:48:05.461361 (+ 195us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":54}
I20260620 14:48:05.461764 7271 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:48:05.450319 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51868 (local address 127.0.97.60:37899)
0620 14:48:05.450474 (+ 155us) server_negotiation.cc:206] Beginning negotiation
0620 14:48:05.450478 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 14:48:05.450620 (+ 142us) server_negotiation.cc:407] Connection header received
0620 14:48:05.450718 (+ 98us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:48:05.450721 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:48:05.450774 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:48:05.450846 (+ 72us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:48:05.451432 (+ 586us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:48:05.452172 (+ 740us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:48:05.453276 (+ 1104us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:48:05.453652 (+ 376us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:48:05.453748 (+ 96us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:48:05.456790 (+ 3042us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 14:48:05.456795 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 14:48:05.456798 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 14:48:05.456824 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0620 14:48:05.459317 (+ 2493us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:48:05.460009 (+ 692us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:48:05.460012 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:48:05.460015 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:48:05.460139 (+ 124us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:48:05.460483 (+ 344us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:48:05.460486 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:48:05.460488 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:48:05.460784 (+ 296us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 14:48:05.460890 (+ 106us) server_negotiation.cc:1035] Waiting for connection context
0620 14:48:05.461247 (+ 357us) server_negotiation.cc:299] Negotiation successful
0620 14:48:05.461482 (+ 235us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":69,"server-negotiator.queue_time_us":54}
I20260620 14:48:05.466813 7265 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e4dec65330ba4aefa643b5c1fcbc31ae: Bootstrap starting.
I20260620 14:48:05.470461 7265 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P e4dec65330ba4aefa643b5c1fcbc31ae: Neither blocks nor log segments found. Creating new log.
I20260620 14:48:05.471513 7265 log.cc:824] T 00000000000000000000000000000000 P e4dec65330ba4aefa643b5c1fcbc31ae: Log is configured to *not* fsync() on all Append() calls
I20260620 14:48:05.474282 7265 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e4dec65330ba4aefa643b5c1fcbc31ae: No bootstrap required, opened a new log
I20260620 14:48:05.482125 7265 raft_consensus.cc:348] T 00000000000000000000000000000000 P e4dec65330ba4aefa643b5c1fcbc31ae [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d663f8b7f54d4fe68ce4af5b87767fed" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 39515 } } peers { permanent_uuid: "55c0ce5da3694b759b5b3113647c3eaa" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 41719 } } peers { permanent_uuid: "e4dec65330ba4aefa643b5c1fcbc31ae" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 37899 } }
I20260620 14:48:05.482491 7265 raft_consensus.cc:374] T 00000000000000000000000000000000 P e4dec65330ba4aefa643b5c1fcbc31ae [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 14:48:05.482636 7265 raft_consensus.cc:729] T 00000000000000000000000000000000 P e4dec65330ba4aefa643b5c1fcbc31ae [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e4dec65330ba4aefa643b5c1fcbc31ae, State: Initialized, Role: FOLLOWER
I20260620 14:48:05.483306 7265 consensus_queue.cc:260] T 00000000000000000000000000000000 P e4dec65330ba4aefa643b5c1fcbc31ae [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d663f8b7f54d4fe68ce4af5b87767fed" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 39515 } } peers { permanent_uuid: "55c0ce5da3694b759b5b3113647c3eaa" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 41719 } } peers { permanent_uuid: "e4dec65330ba4aefa643b5c1fcbc31ae" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 37899 } }
I20260620 14:48:05.484248 7275 sys_catalog.cc:455] T 00000000000000000000000000000000 P e4dec65330ba4aefa643b5c1fcbc31ae [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d663f8b7f54d4fe68ce4af5b87767fed" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 39515 } } peers { permanent_uuid: "55c0ce5da3694b759b5b3113647c3eaa" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 41719 } } peers { permanent_uuid: "e4dec65330ba4aefa643b5c1fcbc31ae" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 37899 } } }
I20260620 14:48:05.484659 7275 sys_catalog.cc:458] T 00000000000000000000000000000000 P e4dec65330ba4aefa643b5c1fcbc31ae [sys.catalog]: This master's current role is: FOLLOWER
I20260620 14:48:05.485427 7265 sys_catalog.cc:564] T 00000000000000000000000000000000 P e4dec65330ba4aefa643b5c1fcbc31ae [sys.catalog]: configured and running, proceeding with master startup.
W20260620 14:48:05.495030 7286 catalog_manager.cc:1558] T 00000000000000000000000000000000 P e4dec65330ba4aefa643b5c1fcbc31ae: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260620 14:48:05.495193 7286 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260620 14:48:05.499222 7112 master_runner.cc:428] Detected that this master e4dec65330ba4aefa643b5c1fcbc31ae is joining an existing cluster
I20260620 14:48:05.499331 7112 master_runner.cc:432] Initiating AddMaster RPC to add 127.0.97.60:37899
I20260620 14:48:05.537515 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskUci3DU/build/asan/bin/kudu pid 6611
I20260620 14:48:05.538267 7271 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:48:05.504700 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51874 (local address 127.0.97.60:37899)
0620 14:48:05.510390 (+ 5690us) server_negotiation.cc:206] Beginning negotiation
0620 14:48:05.510398 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0620 14:48:05.519308 (+ 8910us) server_negotiation.cc:407] Connection header received
0620 14:48:05.519482 (+ 174us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:48:05.519486 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:48:05.519539 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:48:05.519619 (+ 80us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:48:05.522410 (+ 2791us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:48:05.523655 (+ 1245us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:48:05.527423 (+ 3768us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:48:05.528180 (+ 757us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:48:05.528302 (+ 122us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:48:05.530924 (+ 2622us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 14:48:05.530930 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 14:48:05.530935 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 14:48:05.530971 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0620 14:48:05.534864 (+ 3893us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:48:05.535684 (+ 820us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:48:05.535689 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:48:05.535691 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:48:05.535834 (+ 143us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:48:05.536223 (+ 389us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:48:05.536229 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:48:05.536232 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:48:05.536592 (+ 360us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 14:48:05.536752 (+ 160us) server_negotiation.cc:1035] Waiting for connection context
0620 14:48:05.537679 (+ 927us) server_negotiation.cc:299] Negotiation successful
0620 14:48:05.537902 (+ 223us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":67}
I20260620 14:48:05.539254 7294 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:48:05.511864 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.60:37899 (local address 127.0.0.1:51874)
0620 14:48:05.519217 (+ 7353us) negotiation.cc:107] Waiting for socket to connect
0620 14:48:05.519237 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0620 14:48:05.519382 (+ 145us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 14:48:05.519813 (+ 431us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 14:48:05.519817 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 14:48:05.520642 (+ 825us) client_negotiation.cc:189] Negotiated authn=SASL
0620 14:48:05.520973 (+ 331us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:48:05.520984 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:48:05.523912 (+ 2928us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:48:05.523918 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:48:05.527283 (+ 3365us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:48:05.527299 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:48:05.528611 (+ 1312us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:48:05.528619 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:48:05.528822 (+ 203us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:48:05.529584 (+ 762us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 14:48:05.529607 (+ 23us) client_negotiation.cc:656] calling sasl_client_start()
0620 14:48:05.530717 (+ 1110us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 14:48:05.535019 (+ 4302us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 14:48:05.535024 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 14:48:05.535029 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0620 14:48:05.535526 (+ 497us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 14:48:05.535959 (+ 433us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 14:48:05.535964 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 14:48:05.535967 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0620 14:48:05.536114 (+ 147us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 14:48:05.537235 (+ 1121us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 14:48:05.537240 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 14:48:05.537417 (+ 177us) client_negotiation.cc:769] Sending connection context
0620 14:48:05.538498 (+ 1081us) client_negotiation.cc:240] Negotiation successful
0620 14:48:05.538811 (+ 313us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":7192,"thread_start_us":7142,"threads_started":1}
I20260620 14:48:05.544463 7273 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:48:05.503920 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57932 (local address 127.0.97.62:39515)
0620 14:48:05.504164 (+ 244us) server_negotiation.cc:206] Beginning negotiation
0620 14:48:05.504173 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0620 14:48:05.518225 (+ 14052us) server_negotiation.cc:407] Connection header received
0620 14:48:05.518440 (+ 215us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:48:05.518451 (+ 11us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:48:05.518539 (+ 88us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:48:05.518662 (+ 123us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:48:05.520332 (+ 1670us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:48:05.521490 (+ 1158us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:48:05.523741 (+ 2251us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:48:05.524447 (+ 706us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:48:05.524549 (+ 102us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:48:05.532282 (+ 7733us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 14:48:05.532289 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 14:48:05.532294 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 14:48:05.532334 (+ 40us) server_negotiation.cc:922] calling sasl_server_start()
0620 14:48:05.535936 (+ 3602us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:48:05.542014 (+ 6078us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:48:05.542021 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:48:05.542026 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:48:05.542220 (+ 194us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:48:05.542708 (+ 488us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:48:05.542715 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:48:05.542720 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:48:05.543075 (+ 355us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 14:48:05.543244 (+ 169us) server_negotiation.cc:1035] Waiting for connection context
0620 14:48:05.543868 (+ 624us) server_negotiation.cc:299] Negotiation successful
0620 14:48:05.544108 (+ 240us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":72}
I20260620 14:48:05.545885 7293 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:48:05.503294 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:39515 (local address 127.0.0.1:57932)
0620 14:48:05.518047 (+ 14753us) negotiation.cc:107] Waiting for socket to connect
0620 14:48:05.518076 (+ 29us) client_negotiation.cc:174] Beginning negotiation
0620 14:48:05.518291 (+ 215us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 14:48:05.518859 (+ 568us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 14:48:05.518867 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 14:48:05.519630 (+ 763us) client_negotiation.cc:189] Negotiated authn=SASL
0620 14:48:05.520147 (+ 517us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:48:05.520160 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:48:05.521690 (+ 1530us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:48:05.521695 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:48:05.523531 (+ 1836us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:48:05.523544 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:48:05.524888 (+ 1344us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:48:05.524894 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:48:05.525066 (+ 172us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:48:05.526185 (+ 1119us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 14:48:05.526207 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0620 14:48:05.529331 (+ 3124us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 14:48:05.540213 (+ 10882us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 14:48:05.540222 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 14:48:05.540229 (+ 7us) client_negotiation.cc:760] calling sasl_client_step()
0620 14:48:05.540753 (+ 524us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 14:48:05.542353 (+ 1600us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 14:48:05.542357 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 14:48:05.542360 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0620 14:48:05.542572 (+ 212us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 14:48:05.543351 (+ 779us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 14:48:05.543356 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 14:48:05.543553 (+ 197us) client_negotiation.cc:769] Sending connection context
0620 14:48:05.544710 (+ 1157us) client_negotiation.cc:240] Negotiation successful
0620 14:48:05.545139 (+ 429us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":14547,"thread_start_us":15254,"threads_started":1}
I20260620 14:48:05.549882 7270 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:48:05.503293 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48068 (local address 127.0.97.61:41719)
0620 14:48:05.503475 (+ 182us) server_negotiation.cc:206] Beginning negotiation
0620 14:48:05.503483 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0620 14:48:05.519002 (+ 15519us) server_negotiation.cc:407] Connection header received
0620 14:48:05.519466 (+ 464us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:48:05.519472 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:48:05.519546 (+ 74us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:48:05.519651 (+ 105us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:48:05.525417 (+ 5766us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:48:05.526485 (+ 1068us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:48:05.530095 (+ 3610us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:48:05.530714 (+ 619us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:48:05.530807 (+ 93us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:48:05.542376 (+ 11569us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 14:48:05.542384 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 14:48:05.542389 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 14:48:05.542435 (+ 46us) server_negotiation.cc:922] calling sasl_server_start()
0620 14:48:05.546697 (+ 4262us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:48:05.547566 (+ 869us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:48:05.547573 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:48:05.547577 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:48:05.547781 (+ 204us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:48:05.548231 (+ 450us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:48:05.548236 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:48:05.548239 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:48:05.548556 (+ 317us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 14:48:05.548683 (+ 127us) server_negotiation.cc:1035] Waiting for connection context
0620 14:48:05.549277 (+ 594us) server_negotiation.cc:299] Negotiation successful
0620 14:48:05.549508 (+ 231us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":63}
I20260620 14:48:05.550882 7292 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:48:05.503399 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.61:41719 (local address 127.0.0.1:48068)
0620 14:48:05.518885 (+ 15486us) negotiation.cc:107] Waiting for socket to connect
0620 14:48:05.518907 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0620 14:48:05.519302 (+ 395us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 14:48:05.519828 (+ 526us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 14:48:05.519832 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 14:48:05.521761 (+ 1929us) client_negotiation.cc:189] Negotiated authn=SASL
0620 14:48:05.522208 (+ 447us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:48:05.522222 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:48:05.526658 (+ 4436us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:48:05.526663 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:48:05.529918 (+ 3255us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:48:05.529930 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:48:05.531090 (+ 1160us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:48:05.531096 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:48:05.531277 (+ 181us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:48:05.540979 (+ 9702us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 14:48:05.541006 (+ 27us) client_negotiation.cc:656] calling sasl_client_start()
0620 14:48:05.542112 (+ 1106us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 14:48:05.546858 (+ 4746us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 14:48:05.546863 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 14:48:05.546867 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0620 14:48:05.547422 (+ 555us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 14:48:05.547929 (+ 507us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 14:48:05.547935 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 14:48:05.547939 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0620 14:48:05.548106 (+ 167us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 14:48:05.548775 (+ 669us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 14:48:05.548787 (+ 12us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 14:48:05.548984 (+ 197us) client_negotiation.cc:769] Sending connection context
0620 14:48:05.550137 (+ 1153us) client_negotiation.cc:240] Negotiation successful
0620 14:48:05.550442 (+ 305us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":15328,"thread_start_us":14204,"threads_started":1}
Jun 20 14:48:05 dist-test-slave-q9j6 krb5kdc[6505](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781966882, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.0.97.1@KRBTEST.COM
Jun 20 14:48:05 dist-test-slave-q9j6 krb5kdc[6505](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781966882, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.97.62@KRBTEST.COM for oryx/127.0.97.60@KRBTEST.COM
I20260620 14:48:05.562654 7295 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:48:05.538837 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48830 (local address 127.0.97.1:39249)
0620 14:48:05.539472 (+ 635us) server_negotiation.cc:206] Beginning negotiation
0620 14:48:05.539483 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0620 14:48:05.539790 (+ 307us) server_negotiation.cc:407] Connection header received
0620 14:48:05.539862 (+ 72us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:48:05.539871 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:48:05.540176 (+ 305us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:48:05.542358 (+ 2182us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:48:05.542667 (+ 309us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:48:05.544208 (+ 1541us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:48:05.547096 (+ 2888us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:48:05.547988 (+ 892us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:48:05.554208 (+ 6220us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:48:05.554434 (+ 226us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 14:48:05.554442 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 14:48:05.554457 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 14:48:05.554479 (+ 22us) server_negotiation.cc:922] calling sasl_server_start()
0620 14:48:05.558952 (+ 4473us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:48:05.559977 (+ 1025us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:48:05.559989 (+ 12us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:48:05.559994 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:48:05.560150 (+ 156us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:48:05.560648 (+ 498us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:48:05.560654 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:48:05.560657 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:48:05.561124 (+ 467us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 14:48:05.561419 (+ 295us) server_negotiation.cc:1035] Waiting for connection context
0620 14:48:05.561824 (+ 405us) server_negotiation.cc:299] Negotiation successful
0620 14:48:05.562145 (+ 321us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":456,"thread_start_us":883,"threads_started":1}
I20260620 14:48:05.564354 6734 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 14:48:05.569010 7269 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:48:05.551442 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.60:37899 (local address 127.0.0.1:51880)
0620 14:48:05.551759 (+ 317us) negotiation.cc:107] Waiting for socket to connect
0620 14:48:05.551778 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0620 14:48:05.551876 (+ 98us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 14:48:05.552695 (+ 819us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 14:48:05.552702 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 14:48:05.553176 (+ 474us) client_negotiation.cc:189] Negotiated authn=SASL
0620 14:48:05.553381 (+ 205us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:48:05.553395 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:48:05.554844 (+ 1449us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:48:05.554850 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:48:05.556827 (+ 1977us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:48:05.556841 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:48:05.557716 (+ 875us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:48:05.557722 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:48:05.557904 (+ 182us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:48:05.558465 (+ 561us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 14:48:05.558482 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0620 14:48:05.560953 (+ 2471us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 14:48:05.565902 (+ 4949us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 14:48:05.565912 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 14:48:05.565919 (+ 7us) client_negotiation.cc:760] calling sasl_client_step()
0620 14:48:05.566579 (+ 660us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 14:48:05.567107 (+ 528us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 14:48:05.567115 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 14:48:05.567122 (+ 7us) client_negotiation.cc:760] calling sasl_client_step()
0620 14:48:05.567282 (+ 160us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 14:48:05.568005 (+ 723us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 14:48:05.568011 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 14:48:05.568198 (+ 187us) client_negotiation.cc:769] Sending connection context
0620 14:48:05.568323 (+ 125us) client_negotiation.cc:240] Negotiation successful
0620 14:48:05.568619 (+ 296us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":192}
I20260620 14:48:05.570209 7271 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:48:05.552131 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51880 (local address 127.0.97.60:37899)
0620 14:48:05.552280 (+ 149us) server_negotiation.cc:206] Beginning negotiation
0620 14:48:05.552285 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 14:48:05.552306 (+ 21us) server_negotiation.cc:407] Connection header received
0620 14:48:05.552396 (+ 90us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:48:05.552401 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:48:05.552513 (+ 112us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:48:05.552617 (+ 104us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:48:05.553532 (+ 915us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:48:05.554678 (+ 1146us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:48:05.556994 (+ 2316us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:48:05.557547 (+ 553us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:48:05.557675 (+ 128us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:48:05.561205 (+ 3530us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 14:48:05.561212 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 14:48:05.561218 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 14:48:05.561267 (+ 49us) server_negotiation.cc:922] calling sasl_server_start()
0620 14:48:05.565661 (+ 4394us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:48:05.566778 (+ 1117us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:48:05.566785 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:48:05.566791 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:48:05.566953 (+ 162us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:48:05.567426 (+ 473us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:48:05.567431 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:48:05.567435 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:48:05.567830 (+ 395us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 14:48:05.569437 (+ 1607us) server_negotiation.cc:1035] Waiting for connection context
0620 14:48:05.569567 (+ 130us) server_negotiation.cc:299] Negotiation successful
0620 14:48:05.569821 (+ 254us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":54}
I20260620 14:48:05.571864 6549 catalog_manager.cc:6979] Initiating ChangeConfig request to add master 127.0.97.60:37899: tablet_id: "00000000000000000000000000000000"
type: ADD_PEER
server {
permanent_uuid: "e4dec65330ba4aefa643b5c1fcbc31ae"
member_type: NON_VOTER
last_known_addr {
host: "127.0.97.60"
port: 37899
}
attrs {
promote: true
}
}
dest_uuid: "d663f8b7f54d4fe68ce4af5b87767fed"
cas_config_opid_index: 5
I20260620 14:48:05.574227 6549 consensus_queue.cc:237] T 00000000000000000000000000000000 P d663f8b7f54d4fe68ce4af5b87767fed [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "d663f8b7f54d4fe68ce4af5b87767fed" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 39515 } } peers { permanent_uuid: "55c0ce5da3694b759b5b3113647c3eaa" member_type: NON_VOTER last_known_addr { host: "127.0.97.61" port: 41719 } attrs { promote: true } } peers { permanent_uuid: "e4dec65330ba4aefa643b5c1fcbc31ae" member_type: NON_VOTER last_known_addr { host: "127.0.97.60" port: 37899 } attrs { promote: true } }
I20260620 14:48:05.576341 7267 raft_consensus.cc:2932] T 00000000000000000000000000000000 P d663f8b7f54d4fe68ce4af5b87767fed [term 1 LEADER]: Committing config change with OpId 1.6: config changed from index 5 to 6, NON_VOTER e4dec65330ba4aefa643b5c1fcbc31ae (127.0.97.60) added. New config: { opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "d663f8b7f54d4fe68ce4af5b87767fed" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 39515 } } peers { permanent_uuid: "55c0ce5da3694b759b5b3113647c3eaa" member_type: NON_VOTER last_known_addr { host: "127.0.97.61" port: 41719 } attrs { promote: true } } peers { permanent_uuid: "e4dec65330ba4aefa643b5c1fcbc31ae" member_type: NON_VOTER last_known_addr { host: "127.0.97.60" port: 37899 } attrs { promote: true } } }
I20260620 14:48:05.581452 7240 raft_consensus.cc:3037] T 00000000000000000000000000000000 P e4dec65330ba4aefa643b5c1fcbc31ae [term 0 FOLLOWER]: Advancing to term 1
I20260620 14:48:05.585641 7240 raft_consensus.cc:1261] T 00000000000000000000000000000000 P e4dec65330ba4aefa643b5c1fcbc31ae [term 1 FOLLOWER]: Refusing update from remote peer d663f8b7f54d4fe68ce4af5b87767fed: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20260620 14:48:05.597051 7159 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 55c0ce5da3694b759b5b3113647c3eaa [term 1 LEARNER]: Refusing update from remote peer d663f8b7f54d4fe68ce4af5b87767fed: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20260620 14:48:05.599881 7267 catalog_manager.cc:6983] Successfully completed master ChangeConfig request to add master 127.0.97.60:37899
I20260620 14:48:05.600205 7267 consensus_queue.cc:1035] T 00000000000000000000000000000000 P d663f8b7f54d4fe68ce4af5b87767fed [LEADER]: Connected to new peer: Peer: permanent_uuid: "e4dec65330ba4aefa643b5c1fcbc31ae" member_type: NON_VOTER last_known_addr { host: "127.0.97.60" port: 37899 } attrs { promote: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 14:48:05.604288 7299 consensus_queue.cc:1035] T 00000000000000000000000000000000 P d663f8b7f54d4fe68ce4af5b87767fed [LEADER]: Connected to new peer: Peer: permanent_uuid: "55c0ce5da3694b759b5b3113647c3eaa" member_type: NON_VOTER last_known_addr { host: "127.0.97.61" port: 41719 } attrs { promote: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.002s
I20260620 14:48:05.602433 7298 sys_catalog.cc:455] T 00000000000000000000000000000000 P d663f8b7f54d4fe68ce4af5b87767fed [sys.catalog]: SysCatalogTable state changed. Reason: Config change replication complete. Latest consensus state: current_term: 1 leader_uuid: "d663f8b7f54d4fe68ce4af5b87767fed" committed_config { opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "d663f8b7f54d4fe68ce4af5b87767fed" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 39515 } } peers { permanent_uuid: "55c0ce5da3694b759b5b3113647c3eaa" member_type: NON_VOTER last_known_addr { host: "127.0.97.61" port: 41719 } attrs { promote: true } } peers { permanent_uuid: "e4dec65330ba4aefa643b5c1fcbc31ae" member_type: NON_VOTER last_known_addr { host: "127.0.97.60" port: 37899 } attrs { promote: true } } }
I20260620 14:48:05.611769 7298 sys_catalog.cc:458] T 00000000000000000000000000000000 P d663f8b7f54d4fe68ce4af5b87767fed [sys.catalog]: This master's current role is: LEADER
W20260620 14:48:05.615963 7219 tablet.cc:2369] T 00000000000000000000000000000000 P e4dec65330ba4aefa643b5c1fcbc31ae: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260620 14:48:05.621304 7240 raft_consensus.cc:858] T 00000000000000000000000000000000 P e4dec65330ba4aefa643b5c1fcbc31ae [term 1 NON_PARTICIPANT]: Illegal state: RaftConfig change currently pending. Only one is allowed at a time.
Committed config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d663f8b7f54d4fe68ce4af5b87767fed" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 39515 } } peers { permanent_uuid: "55c0ce5da3694b759b5b3113647c3eaa" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 41719 } } peers { permanent_uuid: "e4dec65330ba4aefa643b5c1fcbc31ae" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 37899 } }.
Pending config: opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "d663f8b7f54d4fe68ce4af5b87767fed" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 39515 } } peers { permanent_uuid: "55c0ce5da3694b759b5b3113647c3eaa" member_type: NON_VOTER last_known_addr { host: "127.0.97.61" port: 41719 } attrs { promote: true } }:
New config: opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "d663f8b7f54d4fe68ce4af5b87767fed" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 39515 } } peers { permanent_uuid: "55c0ce5da3694b759b5b3113647c3eaa" member_type: NON_VOTER last_known_addr { host: "127.0.97.61" port: 41719 } attrs { promote: true } } peers { permanent_uuid: "e4dec65330ba4aefa643b5c1fcbc31ae" member_type: NON_VOTER last_known_addr { host: "127.0.97.60" port: 37899 } attrs { promote: true } }
W20260620 14:48:05.621734 7240 raft_consensus.cc:1565] T 00000000000000000000000000000000 P e4dec65330ba4aefa643b5c1fcbc31ae [term 1 NON_PARTICIPANT]: Could not prepare op 'term: 1 index: 6' and following 0 ops. Status for this op: Illegal state: RaftConfig change currently pending. Only one is allowed at a time.
Committed config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d663f8b7f54d4fe68ce4af5b87767fed" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 39515 } } peers { permanent_uuid: "55c0ce5da3694b759b5b3113647c3eaa" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 41719 } } peers { permanent_uuid: "e4dec65330ba4aefa643b5c1fcbc31ae" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 37899 } }.
Pending config: opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "d663f8b7f54d4fe68ce4af5b87767fed" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 39515 } } peers { permanent_uuid: "55c0ce5da3694b759b5b3113647c3eaa" member_type: NON_VOTER last_known_addr { host: "127.0.97.61" port: 41719 } attrs { promote: true } }:
New config: opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "d663f8b7f54d4fe68ce4af5b87767fed" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 39515 } } peers { permanent_uuid: "55c0ce5da3694b759b5b3113647c3eaa" member_type: NON_VOTER last_known_addr { host: "127.0.97.61" port: 41719 } attrs { promote: true } } peers { permanent_uuid: "e4dec65330ba4aefa643b5c1fcbc31ae" member_type: NON_VOTER last_known_addr { host: "127.0.97.60" port: 37899 } attrs { promote: true } }
I20260620 14:48:05.626036 7112 master.cc:537] Master@127.0.97.60:37899 shutting down...
I20260620 14:48:05.630815 7159 raft_consensus.cc:2932] T 00000000000000000000000000000000 P 55c0ce5da3694b759b5b3113647c3eaa [term 1 LEARNER]: Committing config change with OpId 1.6: config changed from index 5 to 6, NON_VOTER e4dec65330ba4aefa643b5c1fcbc31ae (127.0.97.60) added. New config: { opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "d663f8b7f54d4fe68ce4af5b87767fed" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 39515 } } peers { permanent_uuid: "55c0ce5da3694b759b5b3113647c3eaa" member_type: NON_VOTER last_known_addr { host: "127.0.97.61" port: 41719 } attrs { promote: true } } peers { permanent_uuid: "e4dec65330ba4aefa643b5c1fcbc31ae" member_type: NON_VOTER last_known_addr { host: "127.0.97.60" port: 37899 } attrs { promote: true } } }
I20260620 14:48:05.635113 7187 sys_catalog.cc:455] T 00000000000000000000000000000000 P 55c0ce5da3694b759b5b3113647c3eaa [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "d663f8b7f54d4fe68ce4af5b87767fed" committed_config { opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "d663f8b7f54d4fe68ce4af5b87767fed" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 39515 } } peers { permanent_uuid: "55c0ce5da3694b759b5b3113647c3eaa" member_type: NON_VOTER last_known_addr { host: "127.0.97.61" port: 41719 } attrs { promote: true } } peers { permanent_uuid: "e4dec65330ba4aefa643b5c1fcbc31ae" member_type: NON_VOTER last_known_addr { host: "127.0.97.60" port: 37899 } attrs { promote: true } } }
I20260620 14:48:05.635368 7187 sys_catalog.cc:458] T 00000000000000000000000000000000 P 55c0ce5da3694b759b5b3113647c3eaa [sys.catalog]: This master's current role is: LEARNER
I20260620 14:48:05.637054 7299 raft_consensus.cc:1050] T 00000000000000000000000000000000 P d663f8b7f54d4fe68ce4af5b87767fed: attempting to promote NON_VOTER 55c0ce5da3694b759b5b3113647c3eaa to VOTER
I20260620 14:48:05.638044 7299 consensus_queue.cc:237] T 00000000000000000000000000000000 P d663f8b7f54d4fe68ce4af5b87767fed [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 1.6, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "d663f8b7f54d4fe68ce4af5b87767fed" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 39515 } } peers { permanent_uuid: "55c0ce5da3694b759b5b3113647c3eaa" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 41719 } attrs { promote: false } } peers { permanent_uuid: "e4dec65330ba4aefa643b5c1fcbc31ae" member_type: NON_VOTER last_known_addr { host: "127.0.97.60" port: 37899 } attrs { promote: true } }
I20260620 14:48:05.640875 7159 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 55c0ce5da3694b759b5b3113647c3eaa [term 1 LEARNER]: Refusing update from remote peer d663f8b7f54d4fe68ce4af5b87767fed: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20260620 14:48:05.641748 7267 consensus_queue.cc:1035] T 00000000000000000000000000000000 P d663f8b7f54d4fe68ce4af5b87767fed [LEADER]: Connected to new peer: Peer: permanent_uuid: "55c0ce5da3694b759b5b3113647c3eaa" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 41719 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20260620 14:48:05.644416 7298 raft_consensus.cc:2932] T 00000000000000000000000000000000 P d663f8b7f54d4fe68ce4af5b87767fed [term 1 LEADER]: Committing config change with OpId 1.7: config changed from index 6 to 7, 55c0ce5da3694b759b5b3113647c3eaa (127.0.97.61) changed from NON_VOTER to VOTER. New config: { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "d663f8b7f54d4fe68ce4af5b87767fed" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 39515 } } peers { permanent_uuid: "55c0ce5da3694b759b5b3113647c3eaa" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 41719 } attrs { promote: false } } peers { permanent_uuid: "e4dec65330ba4aefa643b5c1fcbc31ae" member_type: NON_VOTER last_known_addr { host: "127.0.97.60" port: 37899 } attrs { promote: true } } }
I20260620 14:48:05.648240 7298 sys_catalog.cc:455] T 00000000000000000000000000000000 P d663f8b7f54d4fe68ce4af5b87767fed [sys.catalog]: SysCatalogTable state changed. Reason: Config change replication complete. Latest consensus state: current_term: 1 leader_uuid: "d663f8b7f54d4fe68ce4af5b87767fed" committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "d663f8b7f54d4fe68ce4af5b87767fed" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 39515 } } peers { permanent_uuid: "55c0ce5da3694b759b5b3113647c3eaa" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 41719 } attrs { promote: false } } peers { permanent_uuid: "e4dec65330ba4aefa643b5c1fcbc31ae" member_type: NON_VOTER last_known_addr { host: "127.0.97.60" port: 37899 } attrs { promote: true } } }
I20260620 14:48:05.648496 7298 sys_catalog.cc:458] T 00000000000000000000000000000000 P d663f8b7f54d4fe68ce4af5b87767fed [sys.catalog]: This master's current role is: LEADER
I20260620 14:48:05.649313 7267 sys_catalog.cc:455] T 00000000000000000000000000000000 P d663f8b7f54d4fe68ce4af5b87767fed [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "d663f8b7f54d4fe68ce4af5b87767fed" committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "d663f8b7f54d4fe68ce4af5b87767fed" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 39515 } } peers { permanent_uuid: "55c0ce5da3694b759b5b3113647c3eaa" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 41719 } attrs { promote: false } } peers { permanent_uuid: "e4dec65330ba4aefa643b5c1fcbc31ae" member_type: NON_VOTER last_known_addr { host: "127.0.97.60" port: 37899 } attrs { promote: true } } }
I20260620 14:48:05.649536 7267 sys_catalog.cc:458] T 00000000000000000000000000000000 P d663f8b7f54d4fe68ce4af5b87767fed [sys.catalog]: This master's current role is: LEADER
I20260620 14:48:05.650552 7158 raft_consensus.cc:2932] T 00000000000000000000000000000000 P 55c0ce5da3694b759b5b3113647c3eaa [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, 55c0ce5da3694b759b5b3113647c3eaa (127.0.97.61) changed from NON_VOTER to VOTER. New config: { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "d663f8b7f54d4fe68ce4af5b87767fed" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 39515 } } peers { permanent_uuid: "55c0ce5da3694b759b5b3113647c3eaa" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 41719 } attrs { promote: false } } peers { permanent_uuid: "e4dec65330ba4aefa643b5c1fcbc31ae" member_type: NON_VOTER last_known_addr { host: "127.0.97.60" port: 37899 } attrs { promote: true } } }
I20260620 14:48:05.679689 7187 sys_catalog.cc:455] T 00000000000000000000000000000000 P 55c0ce5da3694b759b5b3113647c3eaa [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "d663f8b7f54d4fe68ce4af5b87767fed" committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "d663f8b7f54d4fe68ce4af5b87767fed" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 39515 } } peers { permanent_uuid: "55c0ce5da3694b759b5b3113647c3eaa" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 41719 } attrs { promote: false } } peers { permanent_uuid: "e4dec65330ba4aefa643b5c1fcbc31ae" member_type: NON_VOTER last_known_addr { host: "127.0.97.60" port: 37899 } attrs { promote: true } } }
I20260620 14:48:05.680002 7187 sys_catalog.cc:458] T 00000000000000000000000000000000 P 55c0ce5da3694b759b5b3113647c3eaa [sys.catalog]: This master's current role is: FOLLOWER
W20260620 14:48:05.686204 6536 consensus_peers.cc:487] T 00000000000000000000000000000000 P d663f8b7f54d4fe68ce4af5b87767fed -> Peer e4dec65330ba4aefa643b5c1fcbc31ae (127.0.97.60:37899): Couldn't send request to peer e4dec65330ba4aefa643b5c1fcbc31ae. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on Master. This is attempt 1: this message will repeat every 5th retry.
I20260620 14:48:05.720019 7240 raft_consensus.cc:2932] T 00000000000000000000000000000000 P e4dec65330ba4aefa643b5c1fcbc31ae [term 1 NON_PARTICIPANT]: Committing config change with OpId 1.5: config changed from index -1 to 5, NON_VOTER 55c0ce5da3694b759b5b3113647c3eaa (127.0.97.61) added. New config: { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "d663f8b7f54d4fe68ce4af5b87767fed" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 39515 } } peers { permanent_uuid: "55c0ce5da3694b759b5b3113647c3eaa" member_type: NON_VOTER last_known_addr { host: "127.0.97.61" port: 41719 } attrs { promote: true } } }
I20260620 14:48:05.736071 7309 sys_catalog.cc:455] T 00000000000000000000000000000000 P e4dec65330ba4aefa643b5c1fcbc31ae [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "d663f8b7f54d4fe68ce4af5b87767fed" committed_config { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "d663f8b7f54d4fe68ce4af5b87767fed" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 39515 } } peers { permanent_uuid: "55c0ce5da3694b759b5b3113647c3eaa" member_type: NON_VOTER last_known_addr { host: "127.0.97.61" port: 41719 } attrs { promote: true } } }
I20260620 14:48:05.736486 7309 sys_catalog.cc:458] T 00000000000000000000000000000000 P e4dec65330ba4aefa643b5c1fcbc31ae [sys.catalog]: This master's current role is: NON_PARTICIPANT
I20260620 14:48:05.736642 7275 sys_catalog.cc:455] T 00000000000000000000000000000000 P e4dec65330ba4aefa643b5c1fcbc31ae [sys.catalog]: SysCatalogTable state changed. Reason: New leader d663f8b7f54d4fe68ce4af5b87767fed. Latest consensus state: current_term: 1 leader_uuid: "d663f8b7f54d4fe68ce4af5b87767fed" committed_config { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "d663f8b7f54d4fe68ce4af5b87767fed" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 39515 } } peers { permanent_uuid: "55c0ce5da3694b759b5b3113647c3eaa" member_type: NON_VOTER last_known_addr { host: "127.0.97.61" port: 41719 } attrs { promote: true } } }
I20260620 14:48:05.736837 7275 sys_catalog.cc:458] T 00000000000000000000000000000000 P e4dec65330ba4aefa643b5c1fcbc31ae [sys.catalog]: This master's current role is: NON_PARTICIPANT
I20260620 14:48:05.737162 7306 sys_catalog.cc:455] T 00000000000000000000000000000000 P e4dec65330ba4aefa643b5c1fcbc31ae [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "d663f8b7f54d4fe68ce4af5b87767fed" committed_config { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "d663f8b7f54d4fe68ce4af5b87767fed" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 39515 } } peers { permanent_uuid: "55c0ce5da3694b759b5b3113647c3eaa" member_type: NON_VOTER last_known_addr { host: "127.0.97.61" port: 41719 } attrs { promote: true } } }
I20260620 14:48:05.737454 7306 sys_catalog.cc:458] T 00000000000000000000000000000000 P e4dec65330ba4aefa643b5c1fcbc31ae [sys.catalog]: This master's current role is: NON_PARTICIPANT
I20260620 14:48:05.740782 7112 raft_consensus.cc:2229] T 00000000000000000000000000000000 P e4dec65330ba4aefa643b5c1fcbc31ae [term 1 NON_PARTICIPANT]: Raft consensus shutting down.
I20260620 14:48:05.741253 7112 raft_consensus.cc:2258] T 00000000000000000000000000000000 P e4dec65330ba4aefa643b5c1fcbc31ae [term 1 NON_PARTICIPANT]: Raft consensus is shut down!
I20260620 14:48:05.741433 7112 tablet_replica.cc:330] T 00000000000000000000000000000000 P e4dec65330ba4aefa643b5c1fcbc31ae: stopping tablet replica
I20260620 14:48:05.750442 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskUci3DU/build/asan/bin/kudu with pid 6611
I20260620 14:48:05.751935 7112 master.cc:552] Master@127.0.97.60:37899 shutdown complete.
I20260620 14:48:05.752040 7112 master_runner.cc:305] Clearing existing system tablet
I20260620 14:48:05.752378 7112 fs_manager.cc:359] Metadata directory not provided
I20260620 14:48:05.752501 7112 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 14:48:05.754644 7112 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.002s sys 0.001s
I20260620 14:48:05.776049 7311 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 14:48:05.776614 7112 fs_report.cc:383] FS layout report
--------------------
wal directory:
metadata directory:
1 data directories: /tmp/dist-test-taskUci3DU/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781966829091300-388-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 14:48:05.777143 7112 fs_manager.cc:725] Time spent opening block manager: real 0.022s user 0.010s sys 0.011s
I20260620 14:48:05.777319 7112 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskUci3DU/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781966829091300-388-0/minicluster-data/master-2/data,/tmp/dist-test-taskUci3DU/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781966829091300-388-0/minicluster-data/master-2/wal
uuid: "e4dec65330ba4aefa643b5c1fcbc31ae"
format_stamp: "Formatted at 2026-06-20 14:48:05 on dist-test-slave-q9j6"
server_key: "b2b7f353c82b330cb4f21c9397e5ee9a"
server_key_iv: "f3daaba5e81f9394bbaba79b961423ba"
server_key_version: "encryptionkey@0"
I20260620 14:48:05.778023 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskUci3DU/build/asan/bin/kudu
/tmp/dist-test-taskUci3DU/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskUci3DU/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781966829091300-388-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskUci3DU/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781966829091300-388-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskUci3DU/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781966829091300-388-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskUci3DU/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781966829091300-388-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.1:39249
--local_ip_for_outbound_sockets=127.0.97.1
--tserver_master_addrs=127.0.97.62:39515
--webserver_port=43933
--webserver_interface=127.0.97.1
--keytab_file=/tmp/dist-test-taskUci3DU/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781966829091300-388-0/krb5kdc/oryx_127.0.97.1.keytab
--principal=oryx/127.0.97.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:33997
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_master_addrs=127.0.97.62:39515,127.0.97.61:41719
--tserver_master_addrs=127.0.97.62:39515,127.0.97.61:41719,127.0.97.60:37899 with env {KRB5CCNAME=/tmp/dist-test-taskUci3DU/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781966829091300-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskUci3DU/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781966829091300-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskUci3DU/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781966829091300-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260620 14:48:05.782677 7112 ts_tablet_manager.cc:1905] T 00000000000000000000000000000000 P e4dec65330ba4aefa643b5c1fcbc31ae: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 14:48:05.790076 7112 ts_tablet_manager.cc:1918] T 00000000000000000000000000000000 P e4dec65330ba4aefa643b5c1fcbc31ae: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId unknown
I20260620 14:48:05.790395 7112 log.cc:1196] T 00000000000000000000000000000000 P e4dec65330ba4aefa643b5c1fcbc31ae: Deleting WAL directory at /tmp/dist-test-taskUci3DU/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781966829091300-388-0/minicluster-data/master-2/wal/wals/00000000000000000000000000000000
I20260620 14:48:05.790988 7112 ts_tablet_manager.cc:1939] T 00000000000000000000000000000000 P e4dec65330ba4aefa643b5c1fcbc31ae: Deleting consensus metadata
I20260620 14:48:05.791306 7112 master_runner.cc:315] Copying system tablet from 127.0.97.62:39515
I20260620 14:48:05.828137 7112 tablet_copy_client.cc:316] T 00000000000000000000000000000000 P e4dec65330ba4aefa643b5c1fcbc31ae: tablet copy: Beginning tablet copy session from remote peer at address 127.0.97.62:39515
I20260620 14:48:05.845242 7273 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:48:05.829246 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57942 (local address 127.0.97.62:39515)
0620 14:48:05.829486 (+ 240us) server_negotiation.cc:206] Beginning negotiation
0620 14:48:05.829493 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0620 14:48:05.829979 (+ 486us) server_negotiation.cc:407] Connection header received
0620 14:48:05.830260 (+ 281us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:48:05.830267 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:48:05.830360 (+ 93us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:48:05.830432 (+ 72us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:48:05.831756 (+ 1324us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:48:05.832883 (+ 1127us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:48:05.834865 (+ 1982us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:48:05.835561 (+ 696us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:48:05.835706 (+ 145us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:48:05.838418 (+ 2712us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 14:48:05.838425 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 14:48:05.838430 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 14:48:05.838471 (+ 41us) server_negotiation.cc:922] calling sasl_server_start()
0620 14:48:05.842008 (+ 3537us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:48:05.842929 (+ 921us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:48:05.842936 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:48:05.842939 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:48:05.843089 (+ 150us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:48:05.843546 (+ 457us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:48:05.843552 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:48:05.843556 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:48:05.843969 (+ 413us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 14:48:05.844085 (+ 116us) server_negotiation.cc:1035] Waiting for connection context
0620 14:48:05.844651 (+ 566us) server_negotiation.cc:299] Negotiation successful
0620 14:48:05.844886 (+ 235us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":89}
I20260620 14:48:05.846467 7318 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:48:05.829232 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:39515 (local address 127.0.0.1:57942)
0620 14:48:05.829862 (+ 630us) negotiation.cc:107] Waiting for socket to connect
0620 14:48:05.829893 (+ 31us) client_negotiation.cc:174] Beginning negotiation
0620 14:48:05.830103 (+ 210us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 14:48:05.830648 (+ 545us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 14:48:05.830653 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 14:48:05.831272 (+ 619us) client_negotiation.cc:189] Negotiated authn=SASL
0620 14:48:05.831525 (+ 253us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:48:05.831538 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:48:05.833112 (+ 1574us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:48:05.833117 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:48:05.834664 (+ 1547us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:48:05.834679 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:48:05.836041 (+ 1362us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:48:05.836047 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:48:05.836233 (+ 186us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:48:05.836996 (+ 763us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 14:48:05.837020 (+ 24us) client_negotiation.cc:656] calling sasl_client_start()
0620 14:48:05.838148 (+ 1128us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 14:48:05.842211 (+ 4063us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 14:48:05.842216 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 14:48:05.842221 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0620 14:48:05.842780 (+ 559us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 14:48:05.843238 (+ 458us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 14:48:05.843243 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 14:48:05.843247 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0620 14:48:05.843421 (+ 174us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 14:48:05.844178 (+ 757us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 14:48:05.844183 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 14:48:05.844354 (+ 171us) client_negotiation.cc:769] Sending connection context
0620 14:48:05.845770 (+ 1416us) client_negotiation.cc:240] Negotiation successful
0620 14:48:05.846090 (+ 320us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":476,"thread_start_us":335,"threads_started":1}
I20260620 14:48:05.847407 6570 tablet_copy_service.cc:140] P d663f8b7f54d4fe68ce4af5b87767fed: Received BeginTabletCopySession request for tablet 00000000000000000000000000000000 from peer e4dec65330ba4aefa643b5c1fcbc31ae ({username='oryx', principal='oryx/127.0.97.60@KRBTEST.COM'} at 127.0.0.1:57942)
I20260620 14:48:05.847644 6570 tablet_copy_service.cc:161] P d663f8b7f54d4fe68ce4af5b87767fed: Beginning new tablet copy session on tablet 00000000000000000000000000000000 from peer e4dec65330ba4aefa643b5c1fcbc31ae at {username='oryx', principal='oryx/127.0.97.60@KRBTEST.COM'} at 127.0.0.1:57942: session id = e4dec65330ba4aefa643b5c1fcbc31ae-00000000000000000000000000000000
I20260620 14:48:05.852228 6570 tablet_copy_source_session.cc:205] T 00000000000000000000000000000000 P d663f8b7f54d4fe68ce4af5b87767fed: Tablet Copy: opened 0 blocks and 1 log segments
I20260620 14:48:05.854463 7112 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total
I20260620 14:48:05.863382 7112 tablet_copy_client.cc:791] T 00000000000000000000000000000000 P e4dec65330ba4aefa643b5c1fcbc31ae: tablet copy: Starting download of 0 data blocks...
I20260620 14:48:05.863776 7112 tablet_copy_client.cc:655] T 00000000000000000000000000000000 P e4dec65330ba4aefa643b5c1fcbc31ae: tablet copy: Starting download of 1 WAL segments...
I20260620 14:48:05.866473 7112 tablet_copy_client.cc:531] T 00000000000000000000000000000000 P e4dec65330ba4aefa643b5c1fcbc31ae: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20260620 14:48:05.872054 6570 tablet_copy_service.cc:342] P d663f8b7f54d4fe68ce4af5b87767fed: Request end of tablet copy session e4dec65330ba4aefa643b5c1fcbc31ae-00000000000000000000000000000000 received from {username='oryx', principal='oryx/127.0.97.60@KRBTEST.COM'} at 127.0.0.1:57942
I20260620 14:48:05.872242 6570 tablet_copy_service.cc:433] P d663f8b7f54d4fe68ce4af5b87767fed: ending tablet copy session e4dec65330ba4aefa643b5c1fcbc31ae-00000000000000000000000000000000 on tablet 00000000000000000000000000000000 with peer e4dec65330ba4aefa643b5c1fcbc31ae
I20260620 14:48:05.887974 7112 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 14:48:05.902952 7319 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 14:48:05.913272 7320 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 14:48:05.934592 7112 server_base.cc:1034] running on GCE node
W20260620 14:48:05.936240 7323 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 14:48:05.936782 7112 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 14:48:05.949496 7112 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 14:48:05.950742 7112 hybrid_clock.cc:648] HybridClock initialized: now 1781966885950700 us; error 53 us; skew 500 ppm
I20260620 14:48:05.951092 7112 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 14:48:05 dist-test-slave-q9j6 krb5kdc[6505](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781966885, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.97.60@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 14:48:05.953446 7112 init.cc:377] Logged in from keytab as oryx/127.0.97.60@KRBTEST.COM (short username oryx)
I20260620 14:48:05.976994 7112 webserver.cc:460] Webserver started at http://127.0.97.60:36635/ using document root <none> and password file <none>
I20260620 14:48:05.977633 7112 fs_manager.cc:359] Metadata directory not provided
I20260620 14:48:05.977793 7112 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 14:48:05.987880 7112 fs_manager.cc:709] Time spent opening directory manager: real 0.009s user 0.004s sys 0.004s
I20260620 14:48:05.998421 7329 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 14:48:05.999050 7112 fs_manager.cc:725] Time spent opening block manager: real 0.010s user 0.001s sys 0.009s
I20260620 14:48:05.999262 7112 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskUci3DU/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781966829091300-388-0/minicluster-data/master-2/data,/tmp/dist-test-taskUci3DU/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781966829091300-388-0/minicluster-data/master-2/wal
uuid: "e4dec65330ba4aefa643b5c1fcbc31ae"
format_stamp: "Formatted at 2026-06-20 14:48:05 on dist-test-slave-q9j6"
server_key: "b2b7f353c82b330cb4f21c9397e5ee9a"
server_key_iv: "f3daaba5e81f9394bbaba79b961423ba"
server_key_version: "encryptionkey@0"
I20260620 14:48:05.999548 7112 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskUci3DU/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781966829091300-388-0/minicluster-data/master-2/wal
metadata directory: /tmp/dist-test-taskUci3DU/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781966829091300-388-0/minicluster-data/master-2/wal
1 data directories: /tmp/dist-test-taskUci3DU/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781966829091300-388-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 14:48:06.036039 7112 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 14:48:06.041136 7112 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 14:48:06.076226 7338 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:48:06.070087 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.60:37899 (local address 127.0.0.1:51882)
0620 14:48:06.075754 (+ 5667us) negotiation.cc:107] Waiting for socket to connect
0620 14:48:06.075942 (+ 188us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.60:37899: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":5480,"thread_start_us":5355,"threads_started":1}
I20260620 14:48:06.078774 7338 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:48:06.076983 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.60:37899 (local address 127.0.0.1:51896)
0620 14:48:06.078077 (+ 1094us) negotiation.cc:107] Waiting for socket to connect
0620 14:48:06.078270 (+ 193us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.60:37899: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":910,"threads_started":1}
I20260620 14:48:06.210593 7112 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.60:37899
I20260620 14:48:06.210624 7382 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.60:37899 every 8 connection(s)
I20260620 14:48:06.211551 7112 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskUci3DU/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781966829091300-388-0/minicluster-data/master-2/data/info.pb
I20260620 14:48:06.219470 7383 sys_catalog.cc:263] Verifying existing consensus state
I20260620 14:48:06.224462 7383 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e4dec65330ba4aefa643b5c1fcbc31ae: Bootstrap starting.
I20260620 14:48:06.235754 7383 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e4dec65330ba4aefa643b5c1fcbc31ae: Bootstrap replayed 1/1 log segments. Stats: ops{read=7 overwritten=0 applied=7 ignored=0} inserts{seen=3 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260620 14:48:06.236202 7383 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e4dec65330ba4aefa643b5c1fcbc31ae: Bootstrap complete.
I20260620 14:48:06.237042 7383 raft_consensus.cc:348] T 00000000000000000000000000000000 P e4dec65330ba4aefa643b5c1fcbc31ae [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "d663f8b7f54d4fe68ce4af5b87767fed" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 39515 } } peers { permanent_uuid: "55c0ce5da3694b759b5b3113647c3eaa" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 41719 } attrs { promote: false } } peers { permanent_uuid: "e4dec65330ba4aefa643b5c1fcbc31ae" member_type: NON_VOTER last_known_addr { host: "127.0.97.60" port: 37899 } attrs { promote: true } }
I20260620 14:48:06.237270 7383 raft_consensus.cc:729] T 00000000000000000000000000000000 P e4dec65330ba4aefa643b5c1fcbc31ae [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: e4dec65330ba4aefa643b5c1fcbc31ae, State: Initialized, Role: LEARNER
I20260620 14:48:06.237506 7383 consensus_queue.cc:260] T 00000000000000000000000000000000 P e4dec65330ba4aefa643b5c1fcbc31ae [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 7, Last appended: 1.7, Last appended by leader: 7, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "d663f8b7f54d4fe68ce4af5b87767fed" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 39515 } } peers { permanent_uuid: "55c0ce5da3694b759b5b3113647c3eaa" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 41719 } attrs { promote: false } } peers { permanent_uuid: "e4dec65330ba4aefa643b5c1fcbc31ae" member_type: NON_VOTER last_known_addr { host: "127.0.97.60" port: 37899 } attrs { promote: true } }
I20260620 14:48:06.238332 7386 sys_catalog.cc:455] T 00000000000000000000000000000000 P e4dec65330ba4aefa643b5c1fcbc31ae [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "d663f8b7f54d4fe68ce4af5b87767fed" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 39515 } } peers { permanent_uuid: "55c0ce5da3694b759b5b3113647c3eaa" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 41719 } attrs { promote: false } } peers { permanent_uuid: "e4dec65330ba4aefa643b5c1fcbc31ae" member_type: NON_VOTER last_known_addr { host: "127.0.97.60" port: 37899 } attrs { promote: true } } }
I20260620 14:48:06.238668 7386 sys_catalog.cc:458] T 00000000000000000000000000000000 P e4dec65330ba4aefa643b5c1fcbc31ae [sys.catalog]: This master's current role is: LEARNER
I20260620 14:48:06.238667 7383 sys_catalog.cc:564] T 00000000000000000000000000000000 P e4dec65330ba4aefa643b5c1fcbc31ae [sys.catalog]: configured and running, proceeding with master startup.
I20260620 14:48:06.246908 7397 catalog_manager.cc:1259] Loaded cluster ID: a48bc503bb2347ef82ed8e77d4e21c70
I20260620 14:48:06.246992 7397 catalog_manager.cc:1552] T 00000000000000000000000000000000 P e4dec65330ba4aefa643b5c1fcbc31ae: loading cluster ID for follower catalog manager: success
I20260620 14:48:06.249462 7397 catalog_manager.cc:1574] T 00000000000000000000000000000000 P e4dec65330ba4aefa643b5c1fcbc31ae: acquiring CA information for follower catalog manager: success
I20260620 14:48:06.251931 7397 catalog_manager.cc:1602] T 00000000000000000000000000000000 P e4dec65330ba4aefa643b5c1fcbc31ae: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
W20260620 14:48:06.275952 7312 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 14:48:06.276250 7312 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 14:48:06.276348 7312 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 14:48:06.284699 7312 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 14:48:06.284844 7312 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 14:48:06.284961 7312 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.1
I20260620 14:48:06.295030 7312 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:33997
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskUci3DU/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781966829091300-388-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskUci3DU/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781966829091300-388-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskUci3DU/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781966829091300-388-0/krb5kdc/oryx_127.0.97.1.keytab
--principal=oryx/127.0.97.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.1:39249
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskUci3DU/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781966829091300-388-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.97.1
--webserver_port=43933
--tserver_master_addrs=127.0.97.62:39515,127.0.97.61:41719,127.0.97.60:37899
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.1
--log_dir=/tmp/dist-test-taskUci3DU/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781966829091300-388-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 13:41:20 UTC on ccycloud.quasar-vdagoi.root.comops.site
build id 4457462
ASAN enabled
I20260620 14:48:06.296391 7312 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 14:48:06.297706 7312 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 14:48:06.311707 7402 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 14:48:06.311636 7403 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 14:48:06.311671 7405 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 14:48:06.313020 7312 server_base.cc:1034] running on GCE node
I20260620 14:48:06.313813 7312 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 14:48:06.314857 7312 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 14:48:06.316105 7312 hybrid_clock.cc:648] HybridClock initialized: now 1781966886316054 us; error 45 us; skew 500 ppm
I20260620 14:48:06.316447 7312 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 14:48:06 dist-test-slave-q9j6 krb5kdc[6505](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781966886, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.97.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 14:48:06.320392 7312 init.cc:377] Logged in from keytab as oryx/127.0.97.1@KRBTEST.COM (short username oryx)
I20260620 14:48:06.322033 7312 webserver.cc:460] Webserver started at http://127.0.97.1:43933/ using document root <none> and password file <none>
I20260620 14:48:06.322919 7312 fs_manager.cc:359] Metadata directory not provided
I20260620 14:48:06.323100 7312 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 14:48:06.329245 7312 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.004s sys 0.000s
I20260620 14:48:06.332211 7412 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 14:48:06.333760 7312 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260620 14:48:06.333989 7312 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskUci3DU/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781966829091300-388-0/minicluster-data/ts-0/data,/tmp/dist-test-taskUci3DU/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781966829091300-388-0/minicluster-data/ts-0/wal
uuid: "76303ed5f6ac43b7866d8efd0313bdc2"
format_stamp: "Formatted at 2026-06-20 14:48:03 on dist-test-slave-q9j6"
server_key: "10a533175bd5b8fde16371fb855bfd45"
server_key_iv: "c3995e4aa86737054ed2a52b7554a949"
server_key_version: "encryptionkey@0"
I20260620 14:48:06.335047 7312 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskUci3DU/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781966829091300-388-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskUci3DU/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781966829091300-388-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskUci3DU/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781966829091300-388-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 14:48:06.350014 7312 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 14:48:06.354193 7312 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 14:48:06.354921 7312 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 14:48:06.356009 7312 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 14:48:06.357517 7312 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 14:48:06.357640 7312 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 14:48:06.357746 7312 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 14:48:06.357810 7312 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 14:48:06.395169 7312 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.1:39249
I20260620 14:48:06.395223 7524 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.1:39249 every 8 connection(s)
I20260620 14:48:06.396699 7312 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskUci3DU/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781966829091300-388-0/minicluster-data/ts-0/data/info.pb
I20260620 14:48:06.402127 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskUci3DU/build/asan/bin/kudu as pid 7312
I20260620 14:48:06.402480 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskUci3DU/build/asan/bin/kudu pid 6745
Jun 20 14:48:06 dist-test-slave-q9j6 krb5kdc[6505](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781966886, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.97.1@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM
Jun 20 14:48:06 dist-test-slave-q9j6 krb5kdc[6505](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781966886, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.97.1@KRBTEST.COM for oryx/127.0.97.61@KRBTEST.COM
Jun 20 14:48:06 dist-test-slave-q9j6 krb5kdc[6505](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781966882, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.0.97.2@KRBTEST.COM
Jun 20 14:48:06 dist-test-slave-q9j6 krb5kdc[6505](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781966886, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.97.1@KRBTEST.COM for oryx/127.0.97.60@KRBTEST.COM
I20260620 14:48:06.448110 7530 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:48:06.399909 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.1:35819 (local address 127.0.97.61:41719)
0620 14:48:06.423516 (+ 23607us) server_negotiation.cc:206] Beginning negotiation
0620 14:48:06.423523 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0620 14:48:06.423554 (+ 31us) server_negotiation.cc:407] Connection header received
0620 14:48:06.426010 (+ 2456us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:48:06.426017 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:48:06.426114 (+ 97us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:48:06.426197 (+ 83us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:48:06.429968 (+ 3771us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:48:06.431072 (+ 1104us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:48:06.432962 (+ 1890us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:48:06.433625 (+ 663us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:48:06.433717 (+ 92us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:48:06.439444 (+ 5727us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 14:48:06.439452 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 14:48:06.439460 (+ 8us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 14:48:06.439508 (+ 48us) server_negotiation.cc:922] calling sasl_server_start()
0620 14:48:06.443931 (+ 4423us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:48:06.445256 (+ 1325us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:48:06.445262 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:48:06.445265 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:48:06.445419 (+ 154us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:48:06.446002 (+ 583us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:48:06.446008 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:48:06.446011 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:48:06.446341 (+ 330us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 14:48:06.446478 (+ 137us) server_negotiation.cc:1035] Waiting for connection context
0620 14:48:06.447474 (+ 996us) server_negotiation.cc:299] Negotiation successful
0620 14:48:06.447741 (+ 267us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":23475,"thread_start_us":23676,"threads_started":1}
I20260620 14:48:06.448110 7529 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:48:06.399826 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.1:47177 (local address 127.0.97.62:39515)
0620 14:48:06.424036 (+ 24210us) server_negotiation.cc:206] Beginning negotiation
0620 14:48:06.424043 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0620 14:48:06.425934 (+ 1891us) server_negotiation.cc:407] Connection header received
0620 14:48:06.426199 (+ 265us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:48:06.426204 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:48:06.426282 (+ 78us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:48:06.426363 (+ 81us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:48:06.428782 (+ 2419us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:48:06.429786 (+ 1004us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:48:06.432624 (+ 2838us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:48:06.433256 (+ 632us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:48:06.433374 (+ 118us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:48:06.439960 (+ 6586us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 14:48:06.439967 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 14:48:06.439972 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 14:48:06.440037 (+ 65us) server_negotiation.cc:922] calling sasl_server_start()
0620 14:48:06.444205 (+ 4168us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:48:06.445249 (+ 1044us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:48:06.445256 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:48:06.445260 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:48:06.445408 (+ 148us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:48:06.446368 (+ 960us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:48:06.446373 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:48:06.446376 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:48:06.446726 (+ 350us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 14:48:06.446849 (+ 123us) server_negotiation.cc:1035] Waiting for connection context
0620 14:48:06.447470 (+ 621us) server_negotiation.cc:299] Negotiation successful
0620 14:48:06.447732 (+ 262us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":24088,"thread_start_us":23951,"threads_started":1}
I20260620 14:48:06.448122 7532 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:48:06.400907 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.1:34713 (local address 127.0.97.60:37899)
0620 14:48:06.424450 (+ 23543us) server_negotiation.cc:206] Beginning negotiation
0620 14:48:06.424456 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 14:48:06.425530 (+ 1074us) server_negotiation.cc:407] Connection header received
0620 14:48:06.426103 (+ 573us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:48:06.426109 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:48:06.426201 (+ 92us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:48:06.426308 (+ 107us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:48:06.429938 (+ 3630us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:48:06.431830 (+ 1892us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:48:06.434571 (+ 2741us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:48:06.435219 (+ 648us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:48:06.435349 (+ 130us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:48:06.440055 (+ 4706us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 14:48:06.440064 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 14:48:06.440069 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 14:48:06.440130 (+ 61us) server_negotiation.cc:922] calling sasl_server_start()
0620 14:48:06.443824 (+ 3694us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:48:06.445363 (+ 1539us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:48:06.445368 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:48:06.445372 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:48:06.445514 (+ 142us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:48:06.446039 (+ 525us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:48:06.446044 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:48:06.446049 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:48:06.446484 (+ 435us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 14:48:06.446605 (+ 121us) server_negotiation.cc:1035] Waiting for connection context
0620 14:48:06.447470 (+ 865us) server_negotiation.cc:299] Negotiation successful
0620 14:48:06.447742 (+ 272us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":23423,"thread_start_us":23454,"threads_started":1}
I20260620 14:48:06.448441 7536 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:48:06.404133 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46948 (local address 127.0.97.2:36327)
0620 14:48:06.421529 (+ 17396us) server_negotiation.cc:206] Beginning negotiation
0620 14:48:06.421544 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0620 14:48:06.424705 (+ 3161us) server_negotiation.cc:407] Connection header received
0620 14:48:06.425104 (+ 399us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:48:06.425116 (+ 12us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:48:06.425418 (+ 302us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:48:06.425626 (+ 208us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:48:06.428018 (+ 2392us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:48:06.429293 (+ 1275us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:48:06.431214 (+ 1921us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:48:06.432147 (+ 933us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:48:06.432287 (+ 140us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:48:06.439976 (+ 7689us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 14:48:06.439995 (+ 19us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 14:48:06.440016 (+ 21us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 14:48:06.440079 (+ 63us) server_negotiation.cc:922] calling sasl_server_start()
0620 14:48:06.444378 (+ 4299us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:48:06.445325 (+ 947us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:48:06.445337 (+ 12us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:48:06.445343 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:48:06.445491 (+ 148us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:48:06.445921 (+ 430us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:48:06.445929 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:48:06.445934 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:48:06.446352 (+ 418us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 14:48:06.446615 (+ 263us) server_negotiation.cc:1035] Waiting for connection context
0620 14:48:06.447775 (+ 1160us) server_negotiation.cc:299] Negotiation successful
0620 14:48:06.448028 (+ 253us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":17103,"thread_start_us":17410,"threads_started":1}
I20260620 14:48:06.450054 6868 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 14:48:06.450503 7533 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:48:06.400592 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.60:37899 (local address 127.0.97.1:34713)
0620 14:48:06.425395 (+ 24803us) negotiation.cc:107] Waiting for socket to connect
0620 14:48:06.425416 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0620 14:48:06.425628 (+ 212us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 14:48:06.426702 (+ 1074us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 14:48:06.426714 (+ 12us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 14:48:06.427830 (+ 1116us) client_negotiation.cc:189] Negotiated authn=SASL
0620 14:48:06.429757 (+ 1927us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:48:06.429768 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:48:06.432690 (+ 2922us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:48:06.432697 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:48:06.434414 (+ 1717us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:48:06.434429 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:48:06.435581 (+ 1152us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:48:06.435587 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:48:06.435813 (+ 226us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:48:06.436403 (+ 590us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 14:48:06.436430 (+ 27us) client_negotiation.cc:656] calling sasl_client_start()
0620 14:48:06.439836 (+ 3406us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 14:48:06.444012 (+ 4176us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 14:48:06.444080 (+ 68us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 14:48:06.444102 (+ 22us) client_negotiation.cc:760] calling sasl_client_step()
0620 14:48:06.445081 (+ 979us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 14:48:06.445701 (+ 620us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 14:48:06.445706 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 14:48:06.445710 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0620 14:48:06.445911 (+ 201us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 14:48:06.446732 (+ 821us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 14:48:06.446736 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 14:48:06.447029 (+ 293us) client_negotiation.cc:769] Sending connection context
0620 14:48:06.448359 (+ 1330us) client_negotiation.cc:240] Negotiation successful
0620 14:48:06.449072 (+ 713us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":24624,"mutex_wait_us":2237,"spinlock_wait_cycles":42496,"thread_start_us":23913,"threads_started":1}
I20260620 14:48:06.450515 7531 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:48:06.400592 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.61:41719 (local address 127.0.97.1:35819)
0620 14:48:06.422580 (+ 21988us) negotiation.cc:107] Waiting for socket to connect
0620 14:48:06.422650 (+ 70us) client_negotiation.cc:174] Beginning negotiation
0620 14:48:06.425624 (+ 2974us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 14:48:06.426700 (+ 1076us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 14:48:06.426713 (+ 13us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 14:48:06.427832 (+ 1119us) client_negotiation.cc:189] Negotiated authn=SASL
0620 14:48:06.429785 (+ 1953us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:48:06.429797 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:48:06.431266 (+ 1469us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:48:06.431271 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:48:06.432796 (+ 1525us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:48:06.432809 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:48:06.433985 (+ 1176us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:48:06.433991 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:48:06.434144 (+ 153us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:48:06.434896 (+ 752us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 14:48:06.434923 (+ 27us) client_negotiation.cc:656] calling sasl_client_start()
0620 14:48:06.439113 (+ 4190us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 14:48:06.444124 (+ 5011us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 14:48:06.444130 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 14:48:06.444136 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0620 14:48:06.445104 (+ 968us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 14:48:06.445583 (+ 479us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 14:48:06.445588 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 14:48:06.445591 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0620 14:48:06.445881 (+ 290us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 14:48:06.446580 (+ 699us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 14:48:06.446589 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 14:48:06.447045 (+ 456us) client_negotiation.cc:769] Sending connection context
0620 14:48:06.448422 (+ 1377us) client_negotiation.cc:240] Negotiation successful
0620 14:48:06.448914 (+ 492us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":21616,"mutex_wait_us":1591,"spinlock_wait_cycles":58240,"thread_start_us":21334,"threads_started":1}
I20260620 14:48:06.452092 7527 heartbeater.cc:347] Connected to a master server at 127.0.97.61:41719
I20260620 14:48:06.452567 7527 heartbeater.cc:464] Registering TS with master...
I20260620 14:48:06.452710 7534 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:48:06.401298 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:39515 (local address 127.0.97.1:47177)
0620 14:48:06.425839 (+ 24541us) negotiation.cc:107] Waiting for socket to connect
0620 14:48:06.425862 (+ 23us) client_negotiation.cc:174] Beginning negotiation
0620 14:48:06.426053 (+ 191us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 14:48:06.426700 (+ 647us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 14:48:06.426713 (+ 13us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 14:48:06.427832 (+ 1119us) client_negotiation.cc:189] Negotiated authn=SASL
0620 14:48:06.428566 (+ 734us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:48:06.428587 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:48:06.429946 (+ 1359us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:48:06.429951 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:48:06.432463 (+ 2512us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:48:06.432478 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:48:06.433658 (+ 1180us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:48:06.433680 (+ 22us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:48:06.433980 (+ 300us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:48:06.434729 (+ 749us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 14:48:06.434757 (+ 28us) client_negotiation.cc:656] calling sasl_client_start()
0620 14:48:06.439715 (+ 4958us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 14:48:06.444360 (+ 4645us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 14:48:06.444364 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 14:48:06.444368 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0620 14:48:06.445052 (+ 684us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 14:48:06.445553 (+ 501us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 14:48:06.445559 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 14:48:06.445565 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0620 14:48:06.446191 (+ 626us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 14:48:06.446920 (+ 729us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 14:48:06.446924 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 14:48:06.447029 (+ 105us) client_negotiation.cc:769] Sending connection context
0620 14:48:06.451711 (+ 4682us) client_negotiation.cc:240] Negotiation successful
0620 14:48:06.452364 (+ 653us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":24406,"mutex_wait_us":57,"spinlock_wait_cycles":8320,"thread_start_us":23545,"threads_started":1}
I20260620 14:48:06.453979 7526 heartbeater.cc:347] Connected to a master server at 127.0.97.62:39515
I20260620 14:48:06.454025 7527 heartbeater.cc:511] Master 127.0.97.61:41719 requested a full tablet report, sending...
I20260620 14:48:06.454150 7526 heartbeater.cc:464] Registering TS with master...
I20260620 14:48:06.454535 7526 heartbeater.cc:511] Master 127.0.97.62:39515 requested a full tablet report, sending...
I20260620 14:48:06.454944 7525 heartbeater.cc:347] Connected to a master server at 127.0.97.60:37899
I20260620 14:48:06.455085 7525 heartbeater.cc:464] Registering TS with master...
I20260620 14:48:06.455587 7525 heartbeater.cc:511] Master 127.0.97.60:37899 requested a full tablet report, sending...
I20260620 14:48:06.455993 6549 ts_manager.cc:194] Re-registered known tserver with Master: 76303ed5f6ac43b7866d8efd0313bdc2 (127.0.97.1:39249)
I20260620 14:48:06.456787 7149 ts_manager.cc:194] Registered new tserver with Master: 76303ed5f6ac43b7866d8efd0313bdc2 (127.0.97.1:39249)
I20260620 14:48:06.457114 6549 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.0.97.1@KRBTEST.COM'} at 127.0.97.1:47177
I20260620 14:48:06.457772 7348 ts_manager.cc:194] Registered new tserver with Master: 76303ed5f6ac43b7866d8efd0313bdc2 (127.0.97.1:39249)
I20260620 14:48:06.543967 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskUci3DU/build/asan/bin/kudu with pid 6745
I20260620 14:48:06.561004 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskUci3DU/build/asan/bin/kudu
/tmp/dist-test-taskUci3DU/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskUci3DU/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781966829091300-388-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskUci3DU/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781966829091300-388-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskUci3DU/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781966829091300-388-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskUci3DU/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781966829091300-388-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.2:36327
--local_ip_for_outbound_sockets=127.0.97.2
--tserver_master_addrs=127.0.97.62:39515
--webserver_port=41095
--webserver_interface=127.0.97.2
--keytab_file=/tmp/dist-test-taskUci3DU/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781966829091300-388-0/krb5kdc/oryx_127.0.97.2.keytab
--principal=oryx/127.0.97.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:33997
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_master_addrs=127.0.97.62:39515,127.0.97.61:41719
--tserver_master_addrs=127.0.97.62:39515,127.0.97.61:41719,127.0.97.60:37899 with env {KRB5CCNAME=/tmp/dist-test-taskUci3DU/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781966829091300-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskUci3DU/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781966829091300-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskUci3DU/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781966829091300-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260620 14:48:06.586036 7532 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:48:06.579546 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51910 (local address 127.0.97.60:37899)
0620 14:48:06.579778 (+ 232us) server_negotiation.cc:206] Beginning negotiation
0620 14:48:06.579783 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 14:48:06.579801 (+ 18us) server_negotiation.cc:407] Connection header received
0620 14:48:06.580034 (+ 233us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:48:06.580038 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:48:06.580111 (+ 73us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:48:06.580209 (+ 98us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 14:48:06.580839 (+ 630us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:48:06.581560 (+ 721us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:48:06.584163 (+ 2603us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:48:06.585106 (+ 943us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:48:06.585298 (+ 192us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:48:06.585505 (+ 207us) server_negotiation.cc:1035] Waiting for connection context
0620 14:48:06.585750 (+ 245us) server_negotiation.cc:299] Negotiation successful
0620 14:48:06.585853 (+ 103us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":133}
I20260620 14:48:06.586150 7339 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:48:06.579418 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.60:37899 (local address 127.0.0.1:51910)
0620 14:48:06.579688 (+ 270us) negotiation.cc:107] Waiting for socket to connect
0620 14:48:06.579712 (+ 24us) client_negotiation.cc:174] Beginning negotiation
0620 14:48:06.579884 (+ 172us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 14:48:06.580277 (+ 393us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 14:48:06.580285 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 14:48:06.580321 (+ 36us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 14:48:06.580637 (+ 316us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:48:06.580652 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:48:06.581704 (+ 1052us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:48:06.581710 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:48:06.583896 (+ 2186us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:48:06.583920 (+ 24us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:48:06.585341 (+ 1421us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:48:06.585348 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:48:06.585558 (+ 210us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:48:06.585605 (+ 47us) client_negotiation.cc:769] Sending connection context
0620 14:48:06.585705 (+ 100us) client_negotiation.cc:240] Negotiation successful
0620 14:48:06.585848 (+ 143us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":59}
I20260620 14:48:06.586943 7358 raft_consensus.cc:1203] T 00000000000000000000000000000000 P e4dec65330ba4aefa643b5c1fcbc31ae [term 1 LEARNER]: Deduplicated request from leader. Original: 1.6->[1.7-1.7] Dedup: 1.7->[]
I20260620 14:48:06.587440 7386 sys_catalog.cc:455] T 00000000000000000000000000000000 P e4dec65330ba4aefa643b5c1fcbc31ae [sys.catalog]: SysCatalogTable state changed. Reason: New leader d663f8b7f54d4fe68ce4af5b87767fed. Latest consensus state: current_term: 1 leader_uuid: "d663f8b7f54d4fe68ce4af5b87767fed" committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "d663f8b7f54d4fe68ce4af5b87767fed" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 39515 } } peers { permanent_uuid: "55c0ce5da3694b759b5b3113647c3eaa" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 41719 } attrs { promote: false } } peers { permanent_uuid: "e4dec65330ba4aefa643b5c1fcbc31ae" member_type: NON_VOTER last_known_addr { host: "127.0.97.60" port: 37899 } attrs { promote: true } } }
I20260620 14:48:06.587738 7386 sys_catalog.cc:458] T 00000000000000000000000000000000 P e4dec65330ba4aefa643b5c1fcbc31ae [sys.catalog]: This master's current role is: LEARNER
W20260620 14:48:06.904264 7538 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 14:48:06.904572 7538 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 14:48:06.904738 7538 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 14:48:06.912307 7538 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 14:48:06.912427 7538 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 14:48:06.912636 7538 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.2
I20260620 14:48:06.922830 7538 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:33997
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskUci3DU/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781966829091300-388-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskUci3DU/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781966829091300-388-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskUci3DU/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781966829091300-388-0/krb5kdc/oryx_127.0.97.2.keytab
--principal=oryx/127.0.97.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.2:36327
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskUci3DU/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781966829091300-388-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.97.2
--webserver_port=41095
--tserver_master_addrs=127.0.97.62:39515,127.0.97.61:41719,127.0.97.60:37899
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.2
--log_dir=/tmp/dist-test-taskUci3DU/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781966829091300-388-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 13:41:20 UTC on ccycloud.quasar-vdagoi.root.comops.site
build id 4457462
ASAN enabled
I20260620 14:48:06.924368 7538 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 14:48:06.925817 7538 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 14:48:06.939543 7543 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 14:48:06.939390 7546 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 14:48:06.939582 7544 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 14:48:06.940855 7538 server_base.cc:1034] running on GCE node
I20260620 14:48:06.941504 7538 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 14:48:06.942516 7538 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 14:48:06.943787 7538 hybrid_clock.cc:648] HybridClock initialized: now 1781966886943702 us; error 67 us; skew 500 ppm
I20260620 14:48:06.944126 7538 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 14:48:06 dist-test-slave-q9j6 krb5kdc[6505](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781966886, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.97.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 14:48:06.947469 7538 init.cc:377] Logged in from keytab as oryx/127.0.97.2@KRBTEST.COM (short username oryx)
I20260620 14:48:06.949187 7538 webserver.cc:460] Webserver started at http://127.0.97.2:41095/ using document root <none> and password file <none>
I20260620 14:48:06.949947 7538 fs_manager.cc:359] Metadata directory not provided
I20260620 14:48:06.950114 7538 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 14:48:06.955943 7538 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.003s sys 0.000s
I20260620 14:48:06.958850 7553 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 14:48:06.960469 7538 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260620 14:48:06.960909 7300 sys_catalog.cc:455] T 00000000000000000000000000000000 P d663f8b7f54d4fe68ce4af5b87767fed [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "d663f8b7f54d4fe68ce4af5b87767fed" committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "d663f8b7f54d4fe68ce4af5b87767fed" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 39515 } } peers { permanent_uuid: "55c0ce5da3694b759b5b3113647c3eaa" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 41719 } attrs { promote: false } } peers { permanent_uuid: "e4dec65330ba4aefa643b5c1fcbc31ae" member_type: NON_VOTER last_known_addr { host: "127.0.97.60" port: 37899 } attrs { promote: true } } }
I20260620 14:48:06.961205 7300 sys_catalog.cc:458] T 00000000000000000000000000000000 P d663f8b7f54d4fe68ce4af5b87767fed [sys.catalog]: This master's current role is: LEADER
I20260620 14:48:06.960654 7538 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskUci3DU/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781966829091300-388-0/minicluster-data/ts-1/data,/tmp/dist-test-taskUci3DU/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781966829091300-388-0/minicluster-data/ts-1/wal
uuid: "790f4a4136784937b4bfc2572c01a54b"
format_stamp: "Formatted at 2026-06-20 14:48:03 on dist-test-slave-q9j6"
server_key: "416db6f58c20e48f41471f8b01fcd0d7"
server_key_iv: "98c49a11f1dcbec5107c663c5cd261ca"
server_key_version: "encryptionkey@0"
I20260620 14:48:06.962116 7538 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskUci3DU/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781966829091300-388-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskUci3DU/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781966829091300-388-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskUci3DU/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781966829091300-388-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 14:48:06.962468 7300 raft_consensus.cc:1050] T 00000000000000000000000000000000 P d663f8b7f54d4fe68ce4af5b87767fed: attempting to promote NON_VOTER e4dec65330ba4aefa643b5c1fcbc31ae to VOTER
I20260620 14:48:06.963066 7300 consensus_queue.cc:237] T 00000000000000000000000000000000 P d663f8b7f54d4fe68ce4af5b87767fed [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 1.7, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "d663f8b7f54d4fe68ce4af5b87767fed" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 39515 } } peers { permanent_uuid: "55c0ce5da3694b759b5b3113647c3eaa" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 41719 } attrs { promote: false } } peers { permanent_uuid: "e4dec65330ba4aefa643b5c1fcbc31ae" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 37899 } attrs { promote: false } }
I20260620 14:48:06.965044 7158 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 55c0ce5da3694b759b5b3113647c3eaa [term 1 FOLLOWER]: Refusing update from remote peer d663f8b7f54d4fe68ce4af5b87767fed: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20260620 14:48:06.965106 7358 raft_consensus.cc:1261] T 00000000000000000000000000000000 P e4dec65330ba4aefa643b5c1fcbc31ae [term 1 LEARNER]: Refusing update from remote peer d663f8b7f54d4fe68ce4af5b87767fed: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20260620 14:48:06.965682 7555 consensus_queue.cc:1035] T 00000000000000000000000000000000 P d663f8b7f54d4fe68ce4af5b87767fed [LEADER]: Connected to new peer: Peer: permanent_uuid: "55c0ce5da3694b759b5b3113647c3eaa" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 41719 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20260620 14:48:06.966131 7554 consensus_queue.cc:1035] T 00000000000000000000000000000000 P d663f8b7f54d4fe68ce4af5b87767fed [LEADER]: Connected to new peer: Peer: permanent_uuid: "e4dec65330ba4aefa643b5c1fcbc31ae" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 37899 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20260620 14:48:06.969367 7555 raft_consensus.cc:2932] T 00000000000000000000000000000000 P d663f8b7f54d4fe68ce4af5b87767fed [term 1 LEADER]: Committing config change with OpId 1.8: config changed from index 7 to 8, e4dec65330ba4aefa643b5c1fcbc31ae (127.0.97.60) changed from NON_VOTER to VOTER. New config: { opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "d663f8b7f54d4fe68ce4af5b87767fed" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 39515 } } peers { permanent_uuid: "55c0ce5da3694b759b5b3113647c3eaa" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 41719 } attrs { promote: false } } peers { permanent_uuid: "e4dec65330ba4aefa643b5c1fcbc31ae" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 37899 } attrs { promote: false } } }
I20260620 14:48:06.970264 7158 raft_consensus.cc:2932] T 00000000000000000000000000000000 P 55c0ce5da3694b759b5b3113647c3eaa [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, e4dec65330ba4aefa643b5c1fcbc31ae (127.0.97.60) changed from NON_VOTER to VOTER. New config: { opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "d663f8b7f54d4fe68ce4af5b87767fed" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 39515 } } peers { permanent_uuid: "55c0ce5da3694b759b5b3113647c3eaa" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 41719 } attrs { promote: false } } peers { permanent_uuid: "e4dec65330ba4aefa643b5c1fcbc31ae" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 37899 } attrs { promote: false } } }
I20260620 14:48:06.972307 7555 sys_catalog.cc:455] T 00000000000000000000000000000000 P d663f8b7f54d4fe68ce4af5b87767fed [sys.catalog]: SysCatalogTable state changed. Reason: Config change replication complete. Latest consensus state: current_term: 1 leader_uuid: "d663f8b7f54d4fe68ce4af5b87767fed" committed_config { opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "d663f8b7f54d4fe68ce4af5b87767fed" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 39515 } } peers { permanent_uuid: "55c0ce5da3694b759b5b3113647c3eaa" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 41719 } attrs { promote: false } } peers { permanent_uuid: "e4dec65330ba4aefa643b5c1fcbc31ae" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 37899 } attrs { promote: false } } }
I20260620 14:48:06.972512 7555 sys_catalog.cc:458] T 00000000000000000000000000000000 P d663f8b7f54d4fe68ce4af5b87767fed [sys.catalog]: This master's current role is: LEADER
I20260620 14:48:06.972759 7554 sys_catalog.cc:455] T 00000000000000000000000000000000 P d663f8b7f54d4fe68ce4af5b87767fed [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "d663f8b7f54d4fe68ce4af5b87767fed" committed_config { opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "d663f8b7f54d4fe68ce4af5b87767fed" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 39515 } } peers { permanent_uuid: "55c0ce5da3694b759b5b3113647c3eaa" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 41719 } attrs { promote: false } } peers { permanent_uuid: "e4dec65330ba4aefa643b5c1fcbc31ae" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 37899 } attrs { promote: false } } }
I20260620 14:48:06.973026 7554 sys_catalog.cc:458] T 00000000000000000000000000000000 P d663f8b7f54d4fe68ce4af5b87767fed [sys.catalog]: This master's current role is: LEADER
I20260620 14:48:06.975765 7555 sys_catalog.cc:455] T 00000000000000000000000000000000 P d663f8b7f54d4fe68ce4af5b87767fed [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "d663f8b7f54d4fe68ce4af5b87767fed" committed_config { opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "d663f8b7f54d4fe68ce4af5b87767fed" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 39515 } } peers { permanent_uuid: "55c0ce5da3694b759b5b3113647c3eaa" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 41719 } attrs { promote: false } } peers { permanent_uuid: "e4dec65330ba4aefa643b5c1fcbc31ae" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 37899 } attrs { promote: false } } }
I20260620 14:48:06.976027 7555 sys_catalog.cc:458] T 00000000000000000000000000000000 P d663f8b7f54d4fe68ce4af5b87767fed [sys.catalog]: This master's current role is: LEADER
I20260620 14:48:06.976526 7358 raft_consensus.cc:2932] T 00000000000000000000000000000000 P e4dec65330ba4aefa643b5c1fcbc31ae [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, e4dec65330ba4aefa643b5c1fcbc31ae (127.0.97.60) changed from NON_VOTER to VOTER. New config: { opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "d663f8b7f54d4fe68ce4af5b87767fed" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 39515 } } peers { permanent_uuid: "55c0ce5da3694b759b5b3113647c3eaa" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 41719 } attrs { promote: false } } peers { permanent_uuid: "e4dec65330ba4aefa643b5c1fcbc31ae" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 37899 } attrs { promote: false } } }
I20260620 14:48:06.977167 7562 sys_catalog.cc:455] T 00000000000000000000000000000000 P 55c0ce5da3694b759b5b3113647c3eaa [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "d663f8b7f54d4fe68ce4af5b87767fed" committed_config { opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "d663f8b7f54d4fe68ce4af5b87767fed" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 39515 } } peers { permanent_uuid: "55c0ce5da3694b759b5b3113647c3eaa" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 41719 } attrs { promote: false } } peers { permanent_uuid: "e4dec65330ba4aefa643b5c1fcbc31ae" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 37899 } attrs { promote: false } } }
I20260620 14:48:06.977487 7562 sys_catalog.cc:458] T 00000000000000000000000000000000 P 55c0ce5da3694b759b5b3113647c3eaa [sys.catalog]: This master's current role is: FOLLOWER
I20260620 14:48:06.979794 7386 sys_catalog.cc:455] T 00000000000000000000000000000000 P e4dec65330ba4aefa643b5c1fcbc31ae [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "d663f8b7f54d4fe68ce4af5b87767fed" committed_config { opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "d663f8b7f54d4fe68ce4af5b87767fed" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 39515 } } peers { permanent_uuid: "55c0ce5da3694b759b5b3113647c3eaa" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 41719 } attrs { promote: false } } peers { permanent_uuid: "e4dec65330ba4aefa643b5c1fcbc31ae" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 37899 } attrs { promote: false } } }
I20260620 14:48:06.980083 7386 sys_catalog.cc:458] T 00000000000000000000000000000000 P e4dec65330ba4aefa643b5c1fcbc31ae [sys.catalog]: This master's current role is: FOLLOWER
I20260620 14:48:07.016530 7538 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 14:48:07.021111 7538 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 14:48:07.021449 7538 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 14:48:07.022558 7538 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 14:48:07.024143 7538 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 14:48:07.024256 7538 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 14:48:07.024365 7538 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 14:48:07.024461 7538 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 14:48:07.058158 7538 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.2:36327
I20260620 14:48:07.058218 7672 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.2:36327 every 8 connection(s)
I20260620 14:48:07.059711 7538 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskUci3DU/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781966829091300-388-0/minicluster-data/ts-1/data/info.pb
I20260620 14:48:07.062819 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskUci3DU/build/asan/bin/kudu as pid 7538
I20260620 14:48:07.063069 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskUci3DU/build/asan/bin/kudu pid 6879
Jun 20 14:48:07 dist-test-slave-q9j6 krb5kdc[6505](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781966882, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.0.97.3@KRBTEST.COM
Jun 20 14:48:07 dist-test-slave-q9j6 krb5kdc[6505](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781966886, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.97.2@KRBTEST.COM for oryx/127.0.97.60@KRBTEST.COM
Jun 20 14:48:07 dist-test-slave-q9j6 krb5kdc[6505](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781966886, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.97.2@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM
Jun 20 14:48:07 dist-test-slave-q9j6 krb5kdc[6505](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781966886, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.97.2@KRBTEST.COM for oryx/127.0.97.61@KRBTEST.COM
I20260620 14:48:07.104297 7532 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:48:07.064374 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.2:41209 (local address 127.0.97.60:37899)
0620 14:48:07.064648 (+ 274us) server_negotiation.cc:206] Beginning negotiation
0620 14:48:07.064654 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 14:48:07.078934 (+ 14280us) server_negotiation.cc:407] Connection header received
0620 14:48:07.082285 (+ 3351us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:48:07.082291 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:48:07.082382 (+ 91us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:48:07.082478 (+ 96us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:48:07.085663 (+ 3185us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:48:07.086813 (+ 1150us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:48:07.090277 (+ 3464us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:48:07.091001 (+ 724us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:48:07.091112 (+ 111us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:48:07.095799 (+ 4687us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 14:48:07.095806 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 14:48:07.095812 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 14:48:07.095856 (+ 44us) server_negotiation.cc:922] calling sasl_server_start()
0620 14:48:07.100170 (+ 4314us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:48:07.101303 (+ 1133us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:48:07.101312 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:48:07.101317 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:48:07.101457 (+ 140us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:48:07.102015 (+ 558us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:48:07.102020 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:48:07.102023 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:48:07.102386 (+ 363us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 14:48:07.102514 (+ 128us) server_negotiation.cc:1035] Waiting for connection context
0620 14:48:07.103648 (+ 1134us) server_negotiation.cc:299] Negotiation successful
0620 14:48:07.103907 (+ 259us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":145}
I20260620 14:48:07.104305 7677 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:48:07.062905 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.2:34299 (local address 127.0.97.62:39515)
0620 14:48:07.064194 (+ 1289us) server_negotiation.cc:206] Beginning negotiation
0620 14:48:07.064204 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0620 14:48:07.080460 (+ 16256us) server_negotiation.cc:407] Connection header received
0620 14:48:07.082121 (+ 1661us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:48:07.082127 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:48:07.082225 (+ 98us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:48:07.082341 (+ 116us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:48:07.085640 (+ 3299us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:48:07.086426 (+ 786us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:48:07.090258 (+ 3832us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:48:07.090701 (+ 443us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:48:07.090846 (+ 145us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:48:07.095816 (+ 4970us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 14:48:07.095821 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 14:48:07.095825 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 14:48:07.095859 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0620 14:48:07.100327 (+ 4468us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:48:07.101435 (+ 1108us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:48:07.101441 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:48:07.101444 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:48:07.101601 (+ 157us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:48:07.102055 (+ 454us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:48:07.102060 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:48:07.102065 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:48:07.102425 (+ 360us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 14:48:07.102589 (+ 164us) server_negotiation.cc:1035] Waiting for connection context
0620 14:48:07.103666 (+ 1077us) server_negotiation.cc:299] Negotiation successful
0620 14:48:07.103925 (+ 259us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1114,"thread_start_us":951,"threads_started":1}
I20260620 14:48:07.105015 7678 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:48:07.063289 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.2:49723 (local address 127.0.97.61:41719)
0620 14:48:07.064666 (+ 1377us) server_negotiation.cc:206] Beginning negotiation
0620 14:48:07.064674 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0620 14:48:07.078939 (+ 14265us) server_negotiation.cc:407] Connection header received
0620 14:48:07.083766 (+ 4827us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:48:07.083774 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:48:07.083914 (+ 140us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:48:07.084035 (+ 121us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:48:07.085579 (+ 1544us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:48:07.086813 (+ 1234us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:48:07.091109 (+ 4296us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:48:07.091813 (+ 704us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:48:07.091921 (+ 108us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:48:07.097725 (+ 5804us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 14:48:07.097733 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 14:48:07.097739 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 14:48:07.097789 (+ 50us) server_negotiation.cc:922] calling sasl_server_start()
0620 14:48:07.101448 (+ 3659us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:48:07.102315 (+ 867us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:48:07.102321 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:48:07.102324 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:48:07.102474 (+ 150us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:48:07.102969 (+ 495us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:48:07.102974 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:48:07.102976 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:48:07.103328 (+ 352us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 14:48:07.103463 (+ 135us) server_negotiation.cc:1035] Waiting for connection context
0620 14:48:07.103943 (+ 480us) server_negotiation.cc:299] Negotiation successful
0620 14:48:07.104178 (+ 235us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1252,"thread_start_us":1100,"threads_started":1}
I20260620 14:48:07.105279 7681 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:48:07.064141 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43656 (local address 127.0.97.3:46083)
0620 14:48:07.076301 (+ 12160us) server_negotiation.cc:206] Beginning negotiation
0620 14:48:07.076315 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0620 14:48:07.076361 (+ 46us) server_negotiation.cc:407] Connection header received
0620 14:48:07.076458 (+ 97us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:48:07.076472 (+ 14us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:48:07.076844 (+ 372us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:48:07.077065 (+ 221us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:48:07.081539 (+ 4474us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:48:07.083564 (+ 2025us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:48:07.086602 (+ 3038us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:48:07.088019 (+ 1417us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:48:07.088169 (+ 150us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:48:07.096914 (+ 8745us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 14:48:07.096936 (+ 22us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 14:48:07.096959 (+ 23us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 14:48:07.097013 (+ 54us) server_negotiation.cc:922] calling sasl_server_start()
0620 14:48:07.100937 (+ 3924us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:48:07.102010 (+ 1073us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:48:07.102020 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:48:07.102024 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:48:07.102170 (+ 146us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:48:07.102680 (+ 510us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:48:07.102685 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:48:07.102688 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:48:07.103084 (+ 396us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 14:48:07.103372 (+ 288us) server_negotiation.cc:1035] Waiting for connection context
0620 14:48:07.104577 (+ 1205us) server_negotiation.cc:299] Negotiation successful
0620 14:48:07.104849 (+ 272us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":11919,"thread_start_us":11755,"threads_started":1}
I20260620 14:48:07.106572 7002 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 14:48:07.106662 7683 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:48:07.064769 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:39515 (local address 127.0.97.2:34299)
0620 14:48:07.080337 (+ 15568us) negotiation.cc:107] Waiting for socket to connect
0620 14:48:07.080353 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0620 14:48:07.081796 (+ 1443us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 14:48:07.082817 (+ 1021us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 14:48:07.082832 (+ 15us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 14:48:07.085181 (+ 2349us) client_negotiation.cc:189] Negotiated authn=SASL
0620 14:48:07.085487 (+ 306us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:48:07.085497 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:48:07.086596 (+ 1099us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:48:07.086601 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:48:07.090104 (+ 3503us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:48:07.090121 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:48:07.090939 (+ 818us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:48:07.090946 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:48:07.091247 (+ 301us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:48:07.092029 (+ 782us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 14:48:07.092061 (+ 32us) client_negotiation.cc:656] calling sasl_client_start()
0620 14:48:07.095549 (+ 3488us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 14:48:07.100510 (+ 4961us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 14:48:07.100519 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 14:48:07.100523 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0620 14:48:07.101286 (+ 763us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 14:48:07.101755 (+ 469us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 14:48:07.101759 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 14:48:07.101763 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0620 14:48:07.101925 (+ 162us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 14:48:07.102663 (+ 738us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 14:48:07.102669 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 14:48:07.103172 (+ 503us) client_negotiation.cc:769] Sending connection context
0620 14:48:07.104579 (+ 1407us) client_negotiation.cc:240] Negotiation successful
0620 14:48:07.105243 (+ 664us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":15466,"mutex_wait_us":448,"spinlock_wait_cycles":30464,"thread_start_us":14824,"threads_started":1}
I20260620 14:48:07.106810 7679 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:48:07.063416 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.60:37899 (local address 127.0.97.2:41209)
0620 14:48:07.078733 (+ 15317us) negotiation.cc:107] Waiting for socket to connect
0620 14:48:07.078796 (+ 63us) client_negotiation.cc:174] Beginning negotiation
0620 14:48:07.081796 (+ 3000us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 14:48:07.082817 (+ 1021us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 14:48:07.082832 (+ 15us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 14:48:07.083866 (+ 1034us) client_negotiation.cc:189] Negotiated authn=SASL
0620 14:48:07.084874 (+ 1008us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:48:07.084902 (+ 28us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:48:07.086999 (+ 2097us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:48:07.087003 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:48:07.090072 (+ 3069us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:48:07.090092 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:48:07.091410 (+ 1318us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:48:07.091417 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:48:07.091666 (+ 249us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:48:07.092182 (+ 516us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 14:48:07.092207 (+ 25us) client_negotiation.cc:656] calling sasl_client_start()
0620 14:48:07.095450 (+ 3243us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 14:48:07.100359 (+ 4909us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 14:48:07.100371 (+ 12us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 14:48:07.100393 (+ 22us) client_negotiation.cc:760] calling sasl_client_step()
0620 14:48:07.101158 (+ 765us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 14:48:07.101609 (+ 451us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 14:48:07.101612 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 14:48:07.101616 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0620 14:48:07.101891 (+ 275us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 14:48:07.102639 (+ 748us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 14:48:07.102648 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 14:48:07.103183 (+ 535us) client_negotiation.cc:769] Sending connection context
0620 14:48:07.104563 (+ 1380us) client_negotiation.cc:240] Negotiation successful
0620 14:48:07.106295 (+ 1732us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":15096,"mutex_wait_us":1832,"spinlock_wait_cycles":10752,"thread_start_us":12788,"threads_started":1}
I20260620 14:48:07.107690 7682 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:48:07.063416 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.61:41719 (local address 127.0.97.2:49723)
0620 14:48:07.078786 (+ 15370us) negotiation.cc:107] Waiting for socket to connect
0620 14:48:07.078815 (+ 29us) client_negotiation.cc:174] Beginning negotiation
0620 14:48:07.081795 (+ 2980us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 14:48:07.084388 (+ 2593us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 14:48:07.084392 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 14:48:07.084993 (+ 601us) client_negotiation.cc:189] Negotiated authn=SASL
0620 14:48:07.085359 (+ 366us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:48:07.085377 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:48:07.087203 (+ 1826us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:48:07.087210 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:48:07.090104 (+ 2894us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:48:07.090121 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:48:07.092271 (+ 2150us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:48:07.092277 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:48:07.092490 (+ 213us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:48:07.093090 (+ 600us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 14:48:07.093111 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0620 14:48:07.097458 (+ 4347us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 14:48:07.101628 (+ 4170us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 14:48:07.101636 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 14:48:07.101642 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0620 14:48:07.102177 (+ 535us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 14:48:07.102632 (+ 455us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 14:48:07.102639 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 14:48:07.102642 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0620 14:48:07.102844 (+ 202us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 14:48:07.103536 (+ 692us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 14:48:07.103540 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 14:48:07.103684 (+ 144us) client_negotiation.cc:769] Sending connection context
0620 14:48:07.105664 (+ 1980us) client_negotiation.cc:240] Negotiation successful
0620 14:48:07.107289 (+ 1625us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":13730,"mutex_wait_us":1194,"spinlock_wait_cycles":73984,"thread_start_us":14795,"threads_started":1}
I20260620 14:48:07.108342 7674 heartbeater.cc:347] Connected to a master server at 127.0.97.62:39515
I20260620 14:48:07.108683 7674 heartbeater.cc:464] Registering TS with master...
I20260620 14:48:07.109237 7675 heartbeater.cc:347] Connected to a master server at 127.0.97.61:41719
I20260620 14:48:07.109355 7675 heartbeater.cc:464] Registering TS with master...
I20260620 14:48:07.109604 7673 heartbeater.cc:347] Connected to a master server at 127.0.97.60:37899
I20260620 14:48:07.109704 7673 heartbeater.cc:464] Registering TS with master...
I20260620 14:48:07.110165 7675 heartbeater.cc:511] Master 127.0.97.61:41719 requested a full tablet report, sending...
I20260620 14:48:07.111155 7673 heartbeater.cc:511] Master 127.0.97.60:37899 requested a full tablet report, sending...
I20260620 14:48:07.111763 7674 heartbeater.cc:511] Master 127.0.97.62:39515 requested a full tablet report, sending...
I20260620 14:48:07.112242 7149 ts_manager.cc:194] Registered new tserver with Master: 790f4a4136784937b4bfc2572c01a54b (127.0.97.2:36327)
I20260620 14:48:07.112622 7348 ts_manager.cc:194] Registered new tserver with Master: 790f4a4136784937b4bfc2572c01a54b (127.0.97.2:36327)
I20260620 14:48:07.112723 6549 ts_manager.cc:194] Re-registered known tserver with Master: 790f4a4136784937b4bfc2572c01a54b (127.0.97.2:36327)
I20260620 14:48:07.113842 6549 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.0.97.2@KRBTEST.COM'} at 127.0.97.2:34299
I20260620 14:48:07.195873 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskUci3DU/build/asan/bin/kudu with pid 6879
W20260620 14:48:07.211409 6537 connection.cc:535] server connection from 127.0.97.3:55093 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
I20260620 14:48:07.212114 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskUci3DU/build/asan/bin/kudu
/tmp/dist-test-taskUci3DU/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskUci3DU/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781966829091300-388-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskUci3DU/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781966829091300-388-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskUci3DU/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781966829091300-388-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskUci3DU/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781966829091300-388-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.3:46083
--local_ip_for_outbound_sockets=127.0.97.3
--tserver_master_addrs=127.0.97.62:39515
--webserver_port=34635
--webserver_interface=127.0.97.3
--keytab_file=/tmp/dist-test-taskUci3DU/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781966829091300-388-0/krb5kdc/oryx_127.0.97.3.keytab
--principal=oryx/127.0.97.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:33997
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_master_addrs=127.0.97.62:39515,127.0.97.61:41719
--tserver_master_addrs=127.0.97.62:39515,127.0.97.61:41719,127.0.97.60:37899 with env {KRB5CCNAME=/tmp/dist-test-taskUci3DU/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781966829091300-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskUci3DU/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781966829091300-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskUci3DU/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781966829091300-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260620 14:48:07.459658 7526 heartbeater.cc:503] Master 127.0.97.62:39515 was elected leader, sending a full tablet report...
W20260620 14:48:07.552414 7685 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 14:48:07.552713 7685 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 14:48:07.552902 7685 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 14:48:07.560417 7685 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 14:48:07.560546 7685 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 14:48:07.560767 7685 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.3
I20260620 14:48:07.571154 7685 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:33997
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskUci3DU/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781966829091300-388-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskUci3DU/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781966829091300-388-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskUci3DU/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781966829091300-388-0/krb5kdc/oryx_127.0.97.3.keytab
--principal=oryx/127.0.97.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.3:46083
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskUci3DU/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781966829091300-388-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.97.3
--webserver_port=34635
--tserver_master_addrs=127.0.97.62:39515,127.0.97.61:41719,127.0.97.60:37899
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.3
--log_dir=/tmp/dist-test-taskUci3DU/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781966829091300-388-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 13:41:20 UTC on ccycloud.quasar-vdagoi.root.comops.site
build id 4457462
ASAN enabled
I20260620 14:48:07.572642 7685 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 14:48:07.574048 7685 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 14:48:07.587769 7690 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 14:48:07.587751 7693 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 14:48:07.587805 7691 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 14:48:07.588876 7685 server_base.cc:1034] running on GCE node
I20260620 14:48:07.589499 7685 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 14:48:07.590646 7685 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 14:48:07.591853 7685 hybrid_clock.cc:648] HybridClock initialized: now 1781966887591794 us; error 48 us; skew 500 ppm
I20260620 14:48:07.592211 7685 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 14:48:07 dist-test-slave-q9j6 krb5kdc[6505](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781966887, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.97.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 14:48:07.595543 7685 init.cc:377] Logged in from keytab as oryx/127.0.97.3@KRBTEST.COM (short username oryx)
I20260620 14:48:07.597115 7685 webserver.cc:460] Webserver started at http://127.0.97.3:34635/ using document root <none> and password file <none>
I20260620 14:48:07.597849 7685 fs_manager.cc:359] Metadata directory not provided
I20260620 14:48:07.598018 7685 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 14:48:07.603955 7685 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.003s sys 0.000s
I20260620 14:48:07.607204 7700 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 14:48:07.608729 7685 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260620 14:48:07.608918 7685 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskUci3DU/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781966829091300-388-0/minicluster-data/ts-2/data,/tmp/dist-test-taskUci3DU/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781966829091300-388-0/minicluster-data/ts-2/wal
uuid: "361b2f75b32a48cda1d69fd295208d8d"
format_stamp: "Formatted at 2026-06-20 14:48:04 on dist-test-slave-q9j6"
server_key: "ed5828261852db0a8c4e8a0f8054f519"
server_key_iv: "b801ae422cb0cb97b39aab6d6af7f761"
server_key_version: "encryptionkey@0"
I20260620 14:48:07.609922 7685 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskUci3DU/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781966829091300-388-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskUci3DU/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781966829091300-388-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskUci3DU/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781966829091300-388-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 14:48:07.627583 7685 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 14:48:07.631808 7685 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 14:48:07.632042 7685 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 14:48:07.632973 7685 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 14:48:07.634313 7685 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 14:48:07.634400 7685 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 14:48:07.634479 7685 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 14:48:07.634532 7685 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 14:48:07.667076 7685 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.3:46083
I20260620 14:48:07.667130 7812 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.3:46083 every 8 connection(s)
I20260620 14:48:07.668740 7685 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskUci3DU/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781966829091300-388-0/minicluster-data/ts-2/data/info.pb
I20260620 14:48:07.674443 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskUci3DU/build/asan/bin/kudu as pid 7685
Jun 20 14:48:07 dist-test-slave-q9j6 krb5kdc[6505](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781966887, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.97.3@KRBTEST.COM for oryx/127.0.97.61@KRBTEST.COM
Jun 20 14:48:07 dist-test-slave-q9j6 krb5kdc[6505](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781966887, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.97.3@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM
Jun 20 14:48:07 dist-test-slave-q9j6 krb5kdc[6505](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781966887, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.97.3@KRBTEST.COM for oryx/127.0.97.60@KRBTEST.COM
I20260620 14:48:07.707088 7820 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:48:07.673142 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.3:38289 (local address 127.0.97.61:41719)
0620 14:48:07.680168 (+ 7026us) server_negotiation.cc:206] Beginning negotiation
0620 14:48:07.680177 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0620 14:48:07.685141 (+ 4964us) server_negotiation.cc:407] Connection header received
0620 14:48:07.685340 (+ 199us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:48:07.685345 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:48:07.685434 (+ 89us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:48:07.685545 (+ 111us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:48:07.686929 (+ 1384us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:48:07.687999 (+ 1070us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:48:07.690274 (+ 2275us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:48:07.690781 (+ 507us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:48:07.690929 (+ 148us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:48:07.698297 (+ 7368us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 14:48:07.698304 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 14:48:07.698308 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 14:48:07.698349 (+ 41us) server_negotiation.cc:922] calling sasl_server_start()
0620 14:48:07.702133 (+ 3784us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:48:07.703844 (+ 1711us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:48:07.703854 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:48:07.703860 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:48:07.704078 (+ 218us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:48:07.704753 (+ 675us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:48:07.704762 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:48:07.704767 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:48:07.705169 (+ 402us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 14:48:07.705317 (+ 148us) server_negotiation.cc:1035] Waiting for connection context
0620 14:48:07.706457 (+ 1140us) server_negotiation.cc:299] Negotiation successful
0620 14:48:07.706704 (+ 247us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":6843,"thread_start_us":6716,"threads_started":1}
I20260620 14:48:07.708824 7817 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:48:07.671759 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.3:55521 (local address 127.0.97.62:39515)
0620 14:48:07.680956 (+ 9197us) server_negotiation.cc:206] Beginning negotiation
0620 14:48:07.680961 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 14:48:07.680977 (+ 16us) server_negotiation.cc:407] Connection header received
0620 14:48:07.681036 (+ 59us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:48:07.681039 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:48:07.681103 (+ 64us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:48:07.681253 (+ 150us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:48:07.683098 (+ 1845us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:48:07.683905 (+ 807us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:48:07.686076 (+ 2171us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:48:07.686759 (+ 683us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:48:07.686902 (+ 143us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:48:07.696268 (+ 9366us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 14:48:07.696277 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 14:48:07.696285 (+ 8us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 14:48:07.696333 (+ 48us) server_negotiation.cc:922] calling sasl_server_start()
0620 14:48:07.701924 (+ 5591us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:48:07.705561 (+ 3637us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:48:07.705567 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:48:07.705570 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:48:07.705743 (+ 173us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:48:07.706773 (+ 1030us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:48:07.706778 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:48:07.706782 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:48:07.707138 (+ 356us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 14:48:07.707290 (+ 152us) server_negotiation.cc:1035] Waiting for connection context
0620 14:48:07.708261 (+ 971us) server_negotiation.cc:299] Negotiation successful
0620 14:48:07.708490 (+ 229us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":9085,"thread_start_us":8988,"threads_started":1}
I20260620 14:48:07.709091 7818 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:48:07.672296 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.3:55771 (local address 127.0.97.60:37899)
0620 14:48:07.680593 (+ 8297us) server_negotiation.cc:206] Beginning negotiation
0620 14:48:07.680602 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0620 14:48:07.685956 (+ 5354us) server_negotiation.cc:407] Connection header received
0620 14:48:07.686266 (+ 310us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:48:07.686275 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:48:07.686377 (+ 102us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:48:07.686470 (+ 93us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:48:07.687981 (+ 1511us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:48:07.689130 (+ 1149us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:48:07.691964 (+ 2834us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:48:07.692749 (+ 785us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:48:07.692917 (+ 168us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:48:07.698000 (+ 5083us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 14:48:07.698008 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 14:48:07.698016 (+ 8us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 14:48:07.698062 (+ 46us) server_negotiation.cc:922] calling sasl_server_start()
0620 14:48:07.702712 (+ 4650us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:48:07.703912 (+ 1200us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:48:07.703921 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:48:07.703927 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:48:07.704100 (+ 173us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:48:07.704744 (+ 644us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:48:07.704752 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:48:07.704757 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:48:07.705215 (+ 458us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 14:48:07.705391 (+ 176us) server_negotiation.cc:1035] Waiting for connection context
0620 14:48:07.708271 (+ 2880us) server_negotiation.cc:299] Negotiation successful
0620 14:48:07.708590 (+ 319us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":8143,"thread_start_us":8041,"threads_started":1}
I20260620 14:48:07.710656 7819 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:48:07.672536 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:39515 (local address 127.0.97.3:55521)
0620 14:48:07.673665 (+ 1129us) negotiation.cc:107] Waiting for socket to connect
0620 14:48:07.673740 (+ 75us) client_negotiation.cc:174] Beginning negotiation
0620 14:48:07.677165 (+ 3425us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 14:48:07.681559 (+ 4394us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 14:48:07.681570 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 14:48:07.682250 (+ 680us) client_negotiation.cc:189] Negotiated authn=SASL
0620 14:48:07.682905 (+ 655us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:48:07.682925 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:48:07.684083 (+ 1158us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:48:07.684087 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:48:07.685882 (+ 1795us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:48:07.685900 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:48:07.687221 (+ 1321us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:48:07.687230 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:48:07.687531 (+ 301us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:48:07.691839 (+ 4308us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 14:48:07.691870 (+ 31us) client_negotiation.cc:656] calling sasl_client_start()
0620 14:48:07.695966 (+ 4096us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 14:48:07.704966 (+ 9000us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 14:48:07.704971 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 14:48:07.704974 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0620 14:48:07.705423 (+ 449us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 14:48:07.706460 (+ 1037us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 14:48:07.706466 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 14:48:07.706469 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0620 14:48:07.706628 (+ 159us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 14:48:07.707677 (+ 1049us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 14:48:07.707682 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 14:48:07.707858 (+ 176us) client_negotiation.cc:769] Sending connection context
0620 14:48:07.707984 (+ 126us) client_negotiation.cc:240] Negotiation successful
0620 14:48:07.709673 (+ 1689us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":697,"mutex_wait_us":1343,"thread_start_us":3158,"threads_started":1}
I20260620 14:48:07.712065 7822 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:48:07.672569 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.60:37899 (local address 127.0.97.3:55771)
0620 14:48:07.685823 (+ 13254us) negotiation.cc:107] Waiting for socket to connect
0620 14:48:07.685850 (+ 27us) client_negotiation.cc:174] Beginning negotiation
0620 14:48:07.686083 (+ 233us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 14:48:07.686706 (+ 623us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 14:48:07.686711 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 14:48:07.687400 (+ 689us) client_negotiation.cc:189] Negotiated authn=SASL
0620 14:48:07.687801 (+ 401us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:48:07.687812 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:48:07.689313 (+ 1501us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:48:07.689318 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:48:07.691470 (+ 2152us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:48:07.691486 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:48:07.692964 (+ 1478us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:48:07.692971 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:48:07.693197 (+ 226us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:48:07.694400 (+ 1203us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 14:48:07.694429 (+ 29us) client_negotiation.cc:656] calling sasl_client_start()
0620 14:48:07.697804 (+ 3375us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 14:48:07.702913 (+ 5109us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 14:48:07.702920 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 14:48:07.702927 (+ 7us) client_negotiation.cc:760] calling sasl_client_step()
0620 14:48:07.703738 (+ 811us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 14:48:07.704244 (+ 506us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 14:48:07.704252 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 14:48:07.704259 (+ 7us) client_negotiation.cc:760] calling sasl_client_step()
0620 14:48:07.704594 (+ 335us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 14:48:07.705458 (+ 864us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 14:48:07.705466 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 14:48:07.705898 (+ 432us) client_negotiation.cc:769] Sending connection context
0620 14:48:07.706308 (+ 410us) client_negotiation.cc:240] Negotiation successful
0620 14:48:07.707038 (+ 730us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":13087,"mutex_wait_us":1086,"spinlock_wait_cycles":9984,"thread_start_us":11866,"threads_started":1}
I20260620 14:48:07.715167 7814 heartbeater.cc:347] Connected to a master server at 127.0.97.62:39515
I20260620 14:48:07.715479 7821 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:48:07.672537 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.61:41719 (local address 127.0.97.3:38289)
0620 14:48:07.684981 (+ 12444us) negotiation.cc:107] Waiting for socket to connect
0620 14:48:07.685009 (+ 28us) client_negotiation.cc:174] Beginning negotiation
0620 14:48:07.685184 (+ 175us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 14:48:07.685734 (+ 550us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 14:48:07.685740 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 14:48:07.686430 (+ 690us) client_negotiation.cc:189] Negotiated authn=SASL
0620 14:48:07.686740 (+ 310us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:48:07.686751 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:48:07.688171 (+ 1420us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:48:07.688176 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:48:07.690114 (+ 1938us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:48:07.690126 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:48:07.690972 (+ 846us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:48:07.690978 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:48:07.691169 (+ 191us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:48:07.691860 (+ 691us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 14:48:07.691878 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0620 14:48:07.698054 (+ 6176us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 14:48:07.702443 (+ 4389us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 14:48:07.702458 (+ 15us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 14:48:07.702486 (+ 28us) client_negotiation.cc:760] calling sasl_client_step()
0620 14:48:07.703657 (+ 1171us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 14:48:07.704268 (+ 611us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 14:48:07.704276 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 14:48:07.704282 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0620 14:48:07.704600 (+ 318us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 14:48:07.705415 (+ 815us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 14:48:07.705422 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 14:48:07.705901 (+ 479us) client_negotiation.cc:769] Sending connection context
0620 14:48:07.707852 (+ 1951us) client_negotiation.cc:240] Negotiation successful
0620 14:48:07.708145 (+ 293us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":12238,"mutex_wait_us":784,"spinlock_wait_cycles":96768,"thread_start_us":11203,"threads_started":1}
I20260620 14:48:07.715535 7814 heartbeater.cc:464] Registering TS with master...
I20260620 14:48:07.716881 7814 heartbeater.cc:511] Master 127.0.97.62:39515 requested a full tablet report, sending...
I20260620 14:48:07.718598 6549 ts_manager.cc:194] Re-registered known tserver with Master: 361b2f75b32a48cda1d69fd295208d8d (127.0.97.3:46083)
I20260620 14:48:07.719787 6549 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.0.97.3@KRBTEST.COM'} at 127.0.97.3:55521
I20260620 14:48:07.723809 7815 heartbeater.cc:347] Connected to a master server at 127.0.97.61:41719
I20260620 14:48:07.723949 7815 heartbeater.cc:464] Registering TS with master...
I20260620 14:48:07.724380 7815 heartbeater.cc:511] Master 127.0.97.61:41719 requested a full tablet report, sending...
I20260620 14:48:07.725651 7149 ts_manager.cc:194] Registered new tserver with Master: 361b2f75b32a48cda1d69fd295208d8d (127.0.97.3:46083)
I20260620 14:48:07.728188 7813 heartbeater.cc:347] Connected to a master server at 127.0.97.60:37899
I20260620 14:48:07.728349 7813 heartbeater.cc:464] Registering TS with master...
I20260620 14:48:07.728752 7813 heartbeater.cc:511] Master 127.0.97.60:37899 requested a full tablet report, sending...
I20260620 14:48:07.731292 7348 ts_manager.cc:194] Registered new tserver with Master: 361b2f75b32a48cda1d69fd295208d8d (127.0.97.3:46083)
I20260620 14:48:07.734102 7818 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:48:07.691770 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51926 (local address 127.0.97.60:37899)
0620 14:48:07.709377 (+ 17607us) server_negotiation.cc:206] Beginning negotiation
0620 14:48:07.709385 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0620 14:48:07.709406 (+ 21us) server_negotiation.cc:407] Connection header received
0620 14:48:07.709466 (+ 60us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:48:07.709473 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:48:07.709588 (+ 115us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:48:07.709701 (+ 113us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:48:07.716885 (+ 7184us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:48:07.718609 (+ 1724us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:48:07.720634 (+ 2025us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:48:07.721286 (+ 652us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:48:07.721458 (+ 172us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:48:07.725224 (+ 3766us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 14:48:07.725231 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 14:48:07.725235 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 14:48:07.725274 (+ 39us) server_negotiation.cc:922] calling sasl_server_start()
0620 14:48:07.730509 (+ 5235us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:48:07.732068 (+ 1559us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:48:07.732074 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:48:07.732077 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:48:07.732213 (+ 136us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:48:07.732615 (+ 402us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:48:07.732620 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:48:07.732624 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:48:07.732872 (+ 248us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 14:48:07.733133 (+ 261us) server_negotiation.cc:1035] Waiting for connection context
0620 14:48:07.733484 (+ 351us) server_negotiation.cc:299] Negotiation successful
0620 14:48:07.733697 (+ 213us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":17463,"thread_start_us":35633,"threads_started":1}
I20260620 14:48:07.739176 7820 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:48:07.698489 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48076 (local address 127.0.97.61:41719)
0620 14:48:07.707810 (+ 9321us) server_negotiation.cc:206] Beginning negotiation
0620 14:48:07.707818 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0620 14:48:07.719098 (+ 11280us) server_negotiation.cc:407] Connection header received
0620 14:48:07.719407 (+ 309us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:48:07.719416 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:48:07.719555 (+ 139us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:48:07.719681 (+ 126us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:48:07.723242 (+ 3561us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:48:07.724692 (+ 1450us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:48:07.728915 (+ 4223us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:48:07.729768 (+ 853us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:48:07.729911 (+ 143us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:48:07.732894 (+ 2983us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 14:48:07.732900 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 14:48:07.732905 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 14:48:07.732950 (+ 45us) server_negotiation.cc:922] calling sasl_server_start()
0620 14:48:07.736614 (+ 3664us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:48:07.737248 (+ 634us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:48:07.737253 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:48:07.737257 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:48:07.737388 (+ 131us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:48:07.737773 (+ 385us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:48:07.737778 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:48:07.737781 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:48:07.738054 (+ 273us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 14:48:07.738369 (+ 315us) server_negotiation.cc:1035] Waiting for connection context
0620 14:48:07.738634 (+ 265us) server_negotiation.cc:299] Negotiation successful
0620 14:48:07.738898 (+ 264us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":9199,"thread_start_us":15182,"threads_started":1}
I20260620 14:48:07.741556 7817 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:48:07.691281 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57952 (local address 127.0.97.62:39515)
0620 14:48:07.709066 (+ 17785us) server_negotiation.cc:206] Beginning negotiation
0620 14:48:07.709071 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 14:48:07.714393 (+ 5322us) server_negotiation.cc:407] Connection header received
0620 14:48:07.714645 (+ 252us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:48:07.714649 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:48:07.714735 (+ 86us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:48:07.714800 (+ 65us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:48:07.727319 (+ 12519us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:48:07.728496 (+ 1177us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:48:07.730715 (+ 2219us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:48:07.731396 (+ 681us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:48:07.731526 (+ 130us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:48:07.736041 (+ 4515us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 14:48:07.736047 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 14:48:07.736050 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 14:48:07.736086 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0620 14:48:07.738708 (+ 2622us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:48:07.739631 (+ 923us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:48:07.739640 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:48:07.739645 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:48:07.739779 (+ 134us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:48:07.740165 (+ 386us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:48:07.740169 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:48:07.740171 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:48:07.740512 (+ 341us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 14:48:07.740724 (+ 212us) server_negotiation.cc:1035] Waiting for connection context
0620 14:48:07.741014 (+ 290us) server_negotiation.cc:299] Negotiation successful
0620 14:48:07.741181 (+ 167us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":17694,"thread_start_us":318,"threads_started":1}
I20260620 14:48:07.745944 6549 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:57952:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260620 14:48:07.752705 6549 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260620 14:48:07.800487 7841 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:48:07.770140 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.3:46083 (local address 127.0.0.1:43662)
0620 14:48:07.786566 (+ 16426us) negotiation.cc:107] Waiting for socket to connect
0620 14:48:07.786597 (+ 31us) client_negotiation.cc:174] Beginning negotiation
0620 14:48:07.786785 (+ 188us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 14:48:07.792495 (+ 5710us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 14:48:07.792500 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 14:48:07.792532 (+ 32us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 14:48:07.792907 (+ 375us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:48:07.792925 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:48:07.794491 (+ 1566us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:48:07.794499 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:48:07.797423 (+ 2924us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:48:07.797434 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:48:07.799420 (+ 1986us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:48:07.799427 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:48:07.799636 (+ 209us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:48:07.799672 (+ 36us) client_negotiation.cc:769] Sending connection context
0620 14:48:07.799765 (+ 93us) client_negotiation.cc:240] Negotiation successful
0620 14:48:07.799919 (+ 154us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":16229,"thread_start_us":16056,"threads_started":1}
I20260620 14:48:07.800612 7843 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:48:07.770462 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43662 (local address 127.0.97.3:46083)
0620 14:48:07.791889 (+ 21427us) server_negotiation.cc:206] Beginning negotiation
0620 14:48:07.791895 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 14:48:07.791927 (+ 32us) server_negotiation.cc:407] Connection header received
0620 14:48:07.792017 (+ 90us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:48:07.792023 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:48:07.792255 (+ 232us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:48:07.792507 (+ 252us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 14:48:07.793250 (+ 743us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:48:07.794323 (+ 1073us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:48:07.797577 (+ 3254us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:48:07.799183 (+ 1606us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:48:07.799334 (+ 151us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:48:07.799553 (+ 219us) server_negotiation.cc:1035] Waiting for connection context
0620 14:48:07.800097 (+ 544us) server_negotiation.cc:299] Negotiation successful
0620 14:48:07.800270 (+ 173us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":21273,"thread_start_us":21052,"threads_started":1}
I20260620 14:48:07.803467 7844 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:48:07.770505 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.2:36327 (local address 127.0.0.1:46960)
0620 14:48:07.794998 (+ 24493us) negotiation.cc:107] Waiting for socket to connect
0620 14:48:07.795021 (+ 23us) client_negotiation.cc:174] Beginning negotiation
0620 14:48:07.795167 (+ 146us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 14:48:07.796407 (+ 1240us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 14:48:07.796412 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 14:48:07.796438 (+ 26us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 14:48:07.796828 (+ 390us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:48:07.796839 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:48:07.798474 (+ 1635us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:48:07.798479 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:48:07.800383 (+ 1904us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:48:07.800398 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:48:07.802620 (+ 2222us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:48:07.802627 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:48:07.802822 (+ 195us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:48:07.802852 (+ 30us) client_negotiation.cc:769] Sending connection context
0620 14:48:07.802945 (+ 93us) client_negotiation.cc:240] Negotiation successful
0620 14:48:07.803078 (+ 133us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":24329,"mutex_wait_us":1,"thread_start_us":25380,"threads_started":1}
I20260620 14:48:07.805027 7846 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:48:07.771310 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46960 (local address 127.0.97.2:36327)
0620 14:48:07.793160 (+ 21850us) server_negotiation.cc:206] Beginning negotiation
0620 14:48:07.793166 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 14:48:07.795166 (+ 2000us) server_negotiation.cc:407] Connection header received
0620 14:48:07.795356 (+ 190us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:48:07.795368 (+ 12us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:48:07.795787 (+ 419us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:48:07.796056 (+ 269us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 14:48:07.797025 (+ 969us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:48:07.798327 (+ 1302us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:48:07.800760 (+ 2433us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:48:07.802280 (+ 1520us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:48:07.802516 (+ 236us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:48:07.802733 (+ 217us) server_negotiation.cc:1035] Waiting for connection context
0620 14:48:07.804556 (+ 1823us) server_negotiation.cc:299] Negotiation successful
0620 14:48:07.804737 (+ 181us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":21705,"thread_start_us":21600,"threads_started":1}
I20260620 14:48:07.805158 7845 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:48:07.771201 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48832 (local address 127.0.97.1:39249)
0620 14:48:07.791305 (+ 20104us) server_negotiation.cc:206] Beginning negotiation
0620 14:48:07.791319 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0620 14:48:07.796323 (+ 5004us) server_negotiation.cc:407] Connection header received
0620 14:48:07.797242 (+ 919us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:48:07.797250 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:48:07.797519 (+ 269us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:48:07.797725 (+ 206us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 14:48:07.798278 (+ 553us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:48:07.799889 (+ 1611us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:48:07.802973 (+ 3084us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:48:07.804084 (+ 1111us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:48:07.804235 (+ 151us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:48:07.804448 (+ 213us) server_negotiation.cc:1035] Waiting for connection context
0620 14:48:07.804703 (+ 255us) server_negotiation.cc:299] Negotiation successful
0620 14:48:07.804864 (+ 161us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":19895,"thread_start_us":19772,"threads_started":1}
I20260620 14:48:07.805413 7842 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:48:07.770937 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:39249 (local address 127.0.0.1:48832)
0620 14:48:07.796182 (+ 25245us) negotiation.cc:107] Waiting for socket to connect
0620 14:48:07.796204 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0620 14:48:07.797021 (+ 817us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 14:48:07.797721 (+ 700us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 14:48:07.797726 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 14:48:07.797754 (+ 28us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 14:48:07.798074 (+ 320us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:48:07.798086 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:48:07.800036 (+ 1950us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:48:07.800043 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:48:07.802814 (+ 2771us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:48:07.802825 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:48:07.804295 (+ 1470us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:48:07.804301 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:48:07.804479 (+ 178us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:48:07.804512 (+ 33us) client_negotiation.cc:769] Sending connection context
0620 14:48:07.804620 (+ 108us) client_negotiation.cc:240] Negotiation successful
0620 14:48:07.804744 (+ 124us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":25116,"mutex_wait_us":3,"thread_start_us":23724,"threads_started":1}
I20260620 14:48:07.805009 7748 tablet_service.cc:1467] Processing CreateTablet for tablet a676dfab9a7c47b2860003164f56d54e (DEFAULT_TABLE table=test-table [id=b0513397257647398cf696e60f8c1af5]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 14:48:07.806469 7748 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a676dfab9a7c47b2860003164f56d54e. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 14:48:07.808691 7608 tablet_service.cc:1467] Processing CreateTablet for tablet a676dfab9a7c47b2860003164f56d54e (DEFAULT_TABLE table=test-table [id=b0513397257647398cf696e60f8c1af5]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 14:48:07.809960 7608 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a676dfab9a7c47b2860003164f56d54e. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 14:48:07.812175 7460 tablet_service.cc:1467] Processing CreateTablet for tablet a676dfab9a7c47b2860003164f56d54e (DEFAULT_TABLE table=test-table [id=b0513397257647398cf696e60f8c1af5]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 14:48:07.813787 7460 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a676dfab9a7c47b2860003164f56d54e. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 14:48:07.822351 7848 tablet_bootstrap.cc:492] T a676dfab9a7c47b2860003164f56d54e P 790f4a4136784937b4bfc2572c01a54b: Bootstrap starting.
I20260620 14:48:07.826243 7848 tablet_bootstrap.cc:654] T a676dfab9a7c47b2860003164f56d54e P 790f4a4136784937b4bfc2572c01a54b: Neither blocks nor log segments found. Creating new log.
I20260620 14:48:07.827303 7848 log.cc:824] T a676dfab9a7c47b2860003164f56d54e P 790f4a4136784937b4bfc2572c01a54b: Log is configured to *not* fsync() on all Append() calls
I20260620 14:48:07.828827 7847 tablet_bootstrap.cc:492] T a676dfab9a7c47b2860003164f56d54e P 361b2f75b32a48cda1d69fd295208d8d: Bootstrap starting.
I20260620 14:48:07.831400 7850 tablet_bootstrap.cc:492] T a676dfab9a7c47b2860003164f56d54e P 76303ed5f6ac43b7866d8efd0313bdc2: Bootstrap starting.
I20260620 14:48:07.833012 7847 tablet_bootstrap.cc:654] T a676dfab9a7c47b2860003164f56d54e P 361b2f75b32a48cda1d69fd295208d8d: Neither blocks nor log segments found. Creating new log.
I20260620 14:48:07.834048 7847 log.cc:824] T a676dfab9a7c47b2860003164f56d54e P 361b2f75b32a48cda1d69fd295208d8d: Log is configured to *not* fsync() on all Append() calls
I20260620 14:48:07.834977 7850 tablet_bootstrap.cc:654] T a676dfab9a7c47b2860003164f56d54e P 76303ed5f6ac43b7866d8efd0313bdc2: Neither blocks nor log segments found. Creating new log.
I20260620 14:48:07.835265 7848 tablet_bootstrap.cc:492] T a676dfab9a7c47b2860003164f56d54e P 790f4a4136784937b4bfc2572c01a54b: No bootstrap required, opened a new log
I20260620 14:48:07.835564 7848 ts_tablet_manager.cc:1397] T a676dfab9a7c47b2860003164f56d54e P 790f4a4136784937b4bfc2572c01a54b: Time spent bootstrapping tablet: real 0.013s user 0.007s sys 0.000s
I20260620 14:48:07.836308 7850 log.cc:824] T a676dfab9a7c47b2860003164f56d54e P 76303ed5f6ac43b7866d8efd0313bdc2: Log is configured to *not* fsync() on all Append() calls
I20260620 14:48:07.837340 7847 tablet_bootstrap.cc:492] T a676dfab9a7c47b2860003164f56d54e P 361b2f75b32a48cda1d69fd295208d8d: No bootstrap required, opened a new log
I20260620 14:48:07.837630 7847 ts_tablet_manager.cc:1397] T a676dfab9a7c47b2860003164f56d54e P 361b2f75b32a48cda1d69fd295208d8d: Time spent bootstrapping tablet: real 0.009s user 0.007s sys 0.000s
I20260620 14:48:07.839982 7850 tablet_bootstrap.cc:492] T a676dfab9a7c47b2860003164f56d54e P 76303ed5f6ac43b7866d8efd0313bdc2: No bootstrap required, opened a new log
I20260620 14:48:07.840310 7850 ts_tablet_manager.cc:1397] T a676dfab9a7c47b2860003164f56d54e P 76303ed5f6ac43b7866d8efd0313bdc2: Time spent bootstrapping tablet: real 0.009s user 0.008s sys 0.000s
I20260620 14:48:07.845933 7847 raft_consensus.cc:348] T a676dfab9a7c47b2860003164f56d54e P 361b2f75b32a48cda1d69fd295208d8d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "361b2f75b32a48cda1d69fd295208d8d" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 46083 } } peers { permanent_uuid: "790f4a4136784937b4bfc2572c01a54b" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 36327 } } peers { permanent_uuid: "76303ed5f6ac43b7866d8efd0313bdc2" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 39249 } }
I20260620 14:48:07.846364 7847 raft_consensus.cc:374] T a676dfab9a7c47b2860003164f56d54e P 361b2f75b32a48cda1d69fd295208d8d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 14:48:07.846457 7847 raft_consensus.cc:729] T a676dfab9a7c47b2860003164f56d54e P 361b2f75b32a48cda1d69fd295208d8d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 361b2f75b32a48cda1d69fd295208d8d, State: Initialized, Role: FOLLOWER
I20260620 14:48:07.847119 7847 consensus_queue.cc:260] T a676dfab9a7c47b2860003164f56d54e P 361b2f75b32a48cda1d69fd295208d8d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "361b2f75b32a48cda1d69fd295208d8d" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 46083 } } peers { permanent_uuid: "790f4a4136784937b4bfc2572c01a54b" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 36327 } } peers { permanent_uuid: "76303ed5f6ac43b7866d8efd0313bdc2" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 39249 } }
I20260620 14:48:07.847095 7848 raft_consensus.cc:348] T a676dfab9a7c47b2860003164f56d54e P 790f4a4136784937b4bfc2572c01a54b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "361b2f75b32a48cda1d69fd295208d8d" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 46083 } } peers { permanent_uuid: "790f4a4136784937b4bfc2572c01a54b" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 36327 } } peers { permanent_uuid: "76303ed5f6ac43b7866d8efd0313bdc2" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 39249 } }
I20260620 14:48:07.847533 7848 raft_consensus.cc:374] T a676dfab9a7c47b2860003164f56d54e P 790f4a4136784937b4bfc2572c01a54b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 14:48:07.847656 7848 raft_consensus.cc:729] T a676dfab9a7c47b2860003164f56d54e P 790f4a4136784937b4bfc2572c01a54b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 790f4a4136784937b4bfc2572c01a54b, State: Initialized, Role: FOLLOWER
I20260620 14:48:07.848244 7848 consensus_queue.cc:260] T a676dfab9a7c47b2860003164f56d54e P 790f4a4136784937b4bfc2572c01a54b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "361b2f75b32a48cda1d69fd295208d8d" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 46083 } } peers { permanent_uuid: "790f4a4136784937b4bfc2572c01a54b" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 36327 } } peers { permanent_uuid: "76303ed5f6ac43b7866d8efd0313bdc2" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 39249 } }
I20260620 14:48:07.849062 7847 ts_tablet_manager.cc:1428] T a676dfab9a7c47b2860003164f56d54e P 361b2f75b32a48cda1d69fd295208d8d: Time spent starting tablet: real 0.011s user 0.007s sys 0.006s
I20260620 14:48:07.849316 7814 heartbeater.cc:503] Master 127.0.97.62:39515 was elected leader, sending a full tablet report...
I20260620 14:48:07.851679 7848 ts_tablet_manager.cc:1428] T a676dfab9a7c47b2860003164f56d54e P 790f4a4136784937b4bfc2572c01a54b: Time spent starting tablet: real 0.016s user 0.005s sys 0.011s
I20260620 14:48:07.852068 7674 heartbeater.cc:503] Master 127.0.97.62:39515 was elected leader, sending a full tablet report...
I20260620 14:48:07.852284 7850 raft_consensus.cc:348] T a676dfab9a7c47b2860003164f56d54e P 76303ed5f6ac43b7866d8efd0313bdc2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "361b2f75b32a48cda1d69fd295208d8d" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 46083 } } peers { permanent_uuid: "790f4a4136784937b4bfc2572c01a54b" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 36327 } } peers { permanent_uuid: "76303ed5f6ac43b7866d8efd0313bdc2" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 39249 } }
I20260620 14:48:07.852736 7850 raft_consensus.cc:374] T a676dfab9a7c47b2860003164f56d54e P 76303ed5f6ac43b7866d8efd0313bdc2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 14:48:07.852838 7850 raft_consensus.cc:729] T a676dfab9a7c47b2860003164f56d54e P 76303ed5f6ac43b7866d8efd0313bdc2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 76303ed5f6ac43b7866d8efd0313bdc2, State: Initialized, Role: FOLLOWER
I20260620 14:48:07.853519 7850 consensus_queue.cc:260] T a676dfab9a7c47b2860003164f56d54e P 76303ed5f6ac43b7866d8efd0313bdc2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "361b2f75b32a48cda1d69fd295208d8d" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 46083 } } peers { permanent_uuid: "790f4a4136784937b4bfc2572c01a54b" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 36327 } } peers { permanent_uuid: "76303ed5f6ac43b7866d8efd0313bdc2" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 39249 } }
I20260620 14:48:07.857599 7850 ts_tablet_manager.cc:1428] T a676dfab9a7c47b2860003164f56d54e P 76303ed5f6ac43b7866d8efd0313bdc2: Time spent starting tablet: real 0.017s user 0.009s sys 0.007s
W20260620 14:48:07.900861 7528 tablet.cc:2369] T a676dfab9a7c47b2860003164f56d54e P 76303ed5f6ac43b7866d8efd0313bdc2: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260620 14:48:07.921850 7816 tablet.cc:2369] T a676dfab9a7c47b2860003164f56d54e P 361b2f75b32a48cda1d69fd295208d8d: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260620 14:48:07.967051 7855 raft_consensus.cc:482] T a676dfab9a7c47b2860003164f56d54e P 76303ed5f6ac43b7866d8efd0313bdc2 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260620 14:48:07.967322 7855 raft_consensus.cc:504] T a676dfab9a7c47b2860003164f56d54e P 76303ed5f6ac43b7866d8efd0313bdc2 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "361b2f75b32a48cda1d69fd295208d8d" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 46083 } } peers { permanent_uuid: "790f4a4136784937b4bfc2572c01a54b" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 36327 } } peers { permanent_uuid: "76303ed5f6ac43b7866d8efd0313bdc2" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 39249 } }
I20260620 14:48:07.969334 7855 leader_election.cc:288] T a676dfab9a7c47b2860003164f56d54e P 76303ed5f6ac43b7866d8efd0313bdc2 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 361b2f75b32a48cda1d69fd295208d8d (127.0.97.3:46083), 790f4a4136784937b4bfc2572c01a54b (127.0.97.2:36327)
I20260620 14:48:07.976032 7846 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:48:07.969776 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.1:43821 (local address 127.0.97.2:36327)
0620 14:48:07.970116 (+ 340us) server_negotiation.cc:206] Beginning negotiation
0620 14:48:07.970122 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 14:48:07.970564 (+ 442us) server_negotiation.cc:407] Connection header received
0620 14:48:07.970732 (+ 168us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:48:07.970738 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:48:07.970813 (+ 75us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:48:07.970903 (+ 90us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 14:48:07.971523 (+ 620us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:48:07.972285 (+ 762us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:48:07.974245 (+ 1960us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:48:07.975005 (+ 760us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:48:07.975209 (+ 204us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:48:07.975305 (+ 96us) server_negotiation.cc:1035] Waiting for connection context
0620 14:48:07.975578 (+ 273us) server_negotiation.cc:299] Negotiation successful
0620 14:48:07.975737 (+ 159us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":162}
I20260620 14:48:07.976262 7857 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:48:07.969556 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.2:36327 (local address 127.0.97.1:43821)
0620 14:48:07.970436 (+ 880us) negotiation.cc:107] Waiting for socket to connect
0620 14:48:07.970455 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0620 14:48:07.970590 (+ 135us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 14:48:07.970937 (+ 347us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 14:48:07.970941 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 14:48:07.970965 (+ 24us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 14:48:07.971378 (+ 413us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:48:07.971392 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:48:07.972431 (+ 1039us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:48:07.972437 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:48:07.974079 (+ 1642us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:48:07.974091 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:48:07.975260 (+ 1169us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:48:07.975266 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:48:07.975417 (+ 151us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:48:07.975440 (+ 23us) client_negotiation.cc:769] Sending connection context
0620 14:48:07.975562 (+ 122us) client_negotiation.cc:240] Negotiation successful
0620 14:48:07.975714 (+ 152us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":703,"mutex_wait_us":78,"thread_start_us":769,"threads_started":1}
I20260620 14:48:07.976903 7843 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:48:07.969666 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.1:39545 (local address 127.0.97.3:46083)
0620 14:48:07.969900 (+ 234us) server_negotiation.cc:206] Beginning negotiation
0620 14:48:07.969908 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0620 14:48:07.970768 (+ 860us) server_negotiation.cc:407] Connection header received
0620 14:48:07.970943 (+ 175us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:48:07.970947 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:48:07.971024 (+ 77us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:48:07.971126 (+ 102us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 14:48:07.971723 (+ 597us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:48:07.972482 (+ 759us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:48:07.974305 (+ 1823us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:48:07.975739 (+ 1434us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:48:07.975902 (+ 163us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:48:07.976007 (+ 105us) server_negotiation.cc:1035] Waiting for connection context
0620 14:48:07.976510 (+ 503us) server_negotiation.cc:299] Negotiation successful
0620 14:48:07.976673 (+ 163us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":66}
I20260620 14:48:07.976958 7856 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:48:07.969570 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.3:46083 (local address 127.0.97.1:39545)
0620 14:48:07.970627 (+ 1057us) negotiation.cc:107] Waiting for socket to connect
0620 14:48:07.970668 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0620 14:48:07.970793 (+ 125us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 14:48:07.971155 (+ 362us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 14:48:07.971158 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 14:48:07.971177 (+ 19us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 14:48:07.971508 (+ 331us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:48:07.971518 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:48:07.972631 (+ 1113us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:48:07.972634 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:48:07.974100 (+ 1466us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:48:07.974111 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:48:07.976167 (+ 2056us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:48:07.976176 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:48:07.976346 (+ 170us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:48:07.976384 (+ 38us) client_negotiation.cc:769] Sending connection context
0620 14:48:07.976484 (+ 100us) client_negotiation.cc:240] Negotiation successful
0620 14:48:07.976619 (+ 135us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":888,"mutex_wait_us":59,"thread_start_us":468,"threads_started":1}
I20260620 14:48:07.977393 7628 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a676dfab9a7c47b2860003164f56d54e" candidate_uuid: "76303ed5f6ac43b7866d8efd0313bdc2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "790f4a4136784937b4bfc2572c01a54b" is_pre_election: true
I20260620 14:48:07.977895 7628 raft_consensus.cc:2454] T a676dfab9a7c47b2860003164f56d54e P 790f4a4136784937b4bfc2572c01a54b [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 76303ed5f6ac43b7866d8efd0313bdc2 in term 0.
I20260620 14:48:07.978672 7415 leader_election.cc:302] T a676dfab9a7c47b2860003164f56d54e P 76303ed5f6ac43b7866d8efd0313bdc2 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 76303ed5f6ac43b7866d8efd0313bdc2, 790f4a4136784937b4bfc2572c01a54b; no voters:
I20260620 14:48:07.979125 7855 raft_consensus.cc:2781] T a676dfab9a7c47b2860003164f56d54e P 76303ed5f6ac43b7866d8efd0313bdc2 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260620 14:48:07.979249 7855 raft_consensus.cc:482] T a676dfab9a7c47b2860003164f56d54e P 76303ed5f6ac43b7866d8efd0313bdc2 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260620 14:48:07.979367 7855 raft_consensus.cc:3037] T a676dfab9a7c47b2860003164f56d54e P 76303ed5f6ac43b7866d8efd0313bdc2 [term 0 FOLLOWER]: Advancing to term 1
I20260620 14:48:07.979494 7768 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a676dfab9a7c47b2860003164f56d54e" candidate_uuid: "76303ed5f6ac43b7866d8efd0313bdc2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "361b2f75b32a48cda1d69fd295208d8d" is_pre_election: true
I20260620 14:48:07.980037 7768 raft_consensus.cc:2454] T a676dfab9a7c47b2860003164f56d54e P 361b2f75b32a48cda1d69fd295208d8d [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 76303ed5f6ac43b7866d8efd0313bdc2 in term 0.
I20260620 14:48:07.981912 7855 raft_consensus.cc:504] T a676dfab9a7c47b2860003164f56d54e P 76303ed5f6ac43b7866d8efd0313bdc2 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "361b2f75b32a48cda1d69fd295208d8d" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 46083 } } peers { permanent_uuid: "790f4a4136784937b4bfc2572c01a54b" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 36327 } } peers { permanent_uuid: "76303ed5f6ac43b7866d8efd0313bdc2" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 39249 } }
I20260620 14:48:07.982609 7855 leader_election.cc:288] T a676dfab9a7c47b2860003164f56d54e P 76303ed5f6ac43b7866d8efd0313bdc2 [CANDIDATE]: Term 1 election: Requested vote from peers 361b2f75b32a48cda1d69fd295208d8d (127.0.97.3:46083), 790f4a4136784937b4bfc2572c01a54b (127.0.97.2:36327)
I20260620 14:48:07.983026 7768 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a676dfab9a7c47b2860003164f56d54e" candidate_uuid: "76303ed5f6ac43b7866d8efd0313bdc2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "361b2f75b32a48cda1d69fd295208d8d"
I20260620 14:48:07.983103 7628 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a676dfab9a7c47b2860003164f56d54e" candidate_uuid: "76303ed5f6ac43b7866d8efd0313bdc2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "790f4a4136784937b4bfc2572c01a54b"
I20260620 14:48:07.983228 7768 raft_consensus.cc:3037] T a676dfab9a7c47b2860003164f56d54e P 361b2f75b32a48cda1d69fd295208d8d [term 0 FOLLOWER]: Advancing to term 1
I20260620 14:48:07.983358 7628 raft_consensus.cc:3037] T a676dfab9a7c47b2860003164f56d54e P 790f4a4136784937b4bfc2572c01a54b [term 0 FOLLOWER]: Advancing to term 1
I20260620 14:48:07.985456 7768 raft_consensus.cc:2454] T a676dfab9a7c47b2860003164f56d54e P 361b2f75b32a48cda1d69fd295208d8d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 76303ed5f6ac43b7866d8efd0313bdc2 in term 1.
I20260620 14:48:07.985641 7628 raft_consensus.cc:2454] T a676dfab9a7c47b2860003164f56d54e P 790f4a4136784937b4bfc2572c01a54b [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 76303ed5f6ac43b7866d8efd0313bdc2 in term 1.
I20260620 14:48:07.985918 7414 leader_election.cc:302] T a676dfab9a7c47b2860003164f56d54e P 76303ed5f6ac43b7866d8efd0313bdc2 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 361b2f75b32a48cda1d69fd295208d8d, 76303ed5f6ac43b7866d8efd0313bdc2; no voters:
I20260620 14:48:07.986292 7855 raft_consensus.cc:2781] T a676dfab9a7c47b2860003164f56d54e P 76303ed5f6ac43b7866d8efd0313bdc2 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 14:48:07.986827 7855 raft_consensus.cc:686] T a676dfab9a7c47b2860003164f56d54e P 76303ed5f6ac43b7866d8efd0313bdc2 [term 1 LEADER]: Becoming Leader. State: Replica: 76303ed5f6ac43b7866d8efd0313bdc2, State: Running, Role: LEADER
I20260620 14:48:07.987385 7855 consensus_queue.cc:237] T a676dfab9a7c47b2860003164f56d54e P 76303ed5f6ac43b7866d8efd0313bdc2 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "361b2f75b32a48cda1d69fd295208d8d" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 46083 } } peers { permanent_uuid: "790f4a4136784937b4bfc2572c01a54b" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 36327 } } peers { permanent_uuid: "76303ed5f6ac43b7866d8efd0313bdc2" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 39249 } }
I20260620 14:48:07.992461 6548 catalog_manager.cc:5495] T a676dfab9a7c47b2860003164f56d54e P 76303ed5f6ac43b7866d8efd0313bdc2 reported cstate change: term changed from 0 to 1, leader changed from <none> to 76303ed5f6ac43b7866d8efd0313bdc2 (127.0.97.1). New cstate: current_term: 1 leader_uuid: "76303ed5f6ac43b7866d8efd0313bdc2" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "361b2f75b32a48cda1d69fd295208d8d" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 46083 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "790f4a4136784937b4bfc2572c01a54b" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 36327 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "76303ed5f6ac43b7866d8efd0313bdc2" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 39249 } health_report { overall_health: HEALTHY } } }
I20260620 14:48:08.016044 7845 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:48:08.010529 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48834 (local address 127.0.97.1:39249)
0620 14:48:08.010729 (+ 200us) server_negotiation.cc:206] Beginning negotiation
0620 14:48:08.010734 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 14:48:08.010881 (+ 147us) server_negotiation.cc:407] Connection header received
0620 14:48:08.011118 (+ 237us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:48:08.011123 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:48:08.011216 (+ 93us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:48:08.011354 (+ 138us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 14:48:08.011774 (+ 420us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:48:08.012496 (+ 722us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:48:08.013973 (+ 1477us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:48:08.014374 (+ 401us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:48:08.014470 (+ 96us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:48:08.014999 (+ 529us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 14:48:08.015547 (+ 548us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0620 14:48:08.015698 (+ 151us) server_negotiation.cc:1035] Waiting for connection context
0620 14:48:08.015776 (+ 78us) server_negotiation.cc:299] Negotiation successful
0620 14:48:08.015843 (+ 67us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":60}
I20260620 14:48:08.025028 7628 raft_consensus.cc:1261] T a676dfab9a7c47b2860003164f56d54e P 790f4a4136784937b4bfc2572c01a54b [term 1 FOLLOWER]: Refusing update from remote peer 76303ed5f6ac43b7866d8efd0313bdc2: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 14:48:08.025025 7768 raft_consensus.cc:1261] T a676dfab9a7c47b2860003164f56d54e P 361b2f75b32a48cda1d69fd295208d8d [term 1 FOLLOWER]: Refusing update from remote peer 76303ed5f6ac43b7866d8efd0313bdc2: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 14:48:08.026006 7858 consensus_queue.cc:1035] T a676dfab9a7c47b2860003164f56d54e P 76303ed5f6ac43b7866d8efd0313bdc2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "790f4a4136784937b4bfc2572c01a54b" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 36327 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 14:48:08.026357 7855 consensus_queue.cc:1035] T a676dfab9a7c47b2860003164f56d54e P 76303ed5f6ac43b7866d8efd0313bdc2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "361b2f75b32a48cda1d69fd295208d8d" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 46083 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 14:48:08.039114 7861 mvcc.cc:205] Tried to move back new op lower bound from 7298936373339623424 to 7298936373200236544. Current Snapshot: MvccSnapshot[applied={T|T < 7298936373339623424}]
I20260620 14:48:08.059520 6548 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:57952:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 14:48:08.059830 6548 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:57952:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 14:48:08.069628 6548 catalog_manager.cc:5782] T 00000000000000000000000000000000 P d663f8b7f54d4fe68ce4af5b87767fed: Sending DeleteTablet for 3 replicas of tablet a676dfab9a7c47b2860003164f56d54e
I20260620 14:48:08.070937 7748 tablet_service.cc:1514] Processing DeleteTablet for tablet a676dfab9a7c47b2860003164f56d54e with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 14:48:08 UTC) from {username='oryx', principal='oryx/127.0.97.62@KRBTEST.COM'} at 127.0.0.1:43662
I20260620 14:48:08.072679 7870 tablet_replica.cc:330] T a676dfab9a7c47b2860003164f56d54e P 361b2f75b32a48cda1d69fd295208d8d: stopping tablet replica
I20260620 14:48:08.073768 7870 raft_consensus.cc:2229] T a676dfab9a7c47b2860003164f56d54e P 361b2f75b32a48cda1d69fd295208d8d [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 14:48:08.073781 7460 tablet_service.cc:1514] Processing DeleteTablet for tablet a676dfab9a7c47b2860003164f56d54e with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 14:48:08 UTC) from {username='oryx', principal='oryx/127.0.97.62@KRBTEST.COM'} at 127.0.0.1:48832
I20260620 14:48:08.074311 7870 raft_consensus.cc:2258] T a676dfab9a7c47b2860003164f56d54e P 361b2f75b32a48cda1d69fd295208d8d [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 14:48:08.075359 7608 tablet_service.cc:1514] Processing DeleteTablet for tablet a676dfab9a7c47b2860003164f56d54e with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 14:48:08 UTC) from {username='oryx', principal='oryx/127.0.97.62@KRBTEST.COM'} at 127.0.0.1:46960
I20260620 14:48:08.076730 7870 ts_tablet_manager.cc:1905] T a676dfab9a7c47b2860003164f56d54e P 361b2f75b32a48cda1d69fd295208d8d: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 14:48:08.086162 7870 ts_tablet_manager.cc:1918] T a676dfab9a7c47b2860003164f56d54e P 361b2f75b32a48cda1d69fd295208d8d: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260620 14:48:08.086356 7870 log.cc:1196] T a676dfab9a7c47b2860003164f56d54e P 361b2f75b32a48cda1d69fd295208d8d: Deleting WAL directory at /tmp/dist-test-taskUci3DU/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781966829091300-388-0/minicluster-data/ts-2/wal/wals/a676dfab9a7c47b2860003164f56d54e
I20260620 14:48:08.086993 7870 ts_tablet_manager.cc:1939] T a676dfab9a7c47b2860003164f56d54e P 361b2f75b32a48cda1d69fd295208d8d: Deleting consensus metadata
I20260620 14:48:08.089354 6535 catalog_manager.cc:4841] TS 361b2f75b32a48cda1d69fd295208d8d (127.0.97.3:46083): tablet a676dfab9a7c47b2860003164f56d54e (table test-table [id=b0513397257647398cf696e60f8c1af5]) successfully deleted
I20260620 14:48:08.091040 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskUci3DU/build/asan/bin/kudu pid 6518
I20260620 14:48:08.092290 6580 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 14:48:08.094733 7872 tablet_replica.cc:330] T a676dfab9a7c47b2860003164f56d54e P 790f4a4136784937b4bfc2572c01a54b: stopping tablet replica
I20260620 14:48:08.095278 7872 raft_consensus.cc:2229] T a676dfab9a7c47b2860003164f56d54e P 790f4a4136784937b4bfc2572c01a54b [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 14:48:08.095991 7872 raft_consensus.cc:2258] T a676dfab9a7c47b2860003164f56d54e P 790f4a4136784937b4bfc2572c01a54b [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 14:48:08.102291 7872 ts_tablet_manager.cc:1905] T a676dfab9a7c47b2860003164f56d54e P 790f4a4136784937b4bfc2572c01a54b: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 14:48:08.110890 7820 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:48:08.077801 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48082 (local address 127.0.97.61:41719)
0620 14:48:08.077989 (+ 188us) server_negotiation.cc:206] Beginning negotiation
0620 14:48:08.077996 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0620 14:48:08.087283 (+ 9287us) server_negotiation.cc:407] Connection header received
0620 14:48:08.087540 (+ 257us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:48:08.087546 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:48:08.087673 (+ 127us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:48:08.087781 (+ 108us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:48:08.090294 (+ 2513us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:48:08.091654 (+ 1360us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:48:08.094555 (+ 2901us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:48:08.095522 (+ 967us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:48:08.095737 (+ 215us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:48:08.101412 (+ 5675us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 14:48:08.101424 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 14:48:08.101432 (+ 8us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 14:48:08.101497 (+ 65us) server_negotiation.cc:922] calling sasl_server_start()
0620 14:48:08.106341 (+ 4844us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:48:08.107743 (+ 1402us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:48:08.107753 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:48:08.107760 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:48:08.107964 (+ 204us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:48:08.108526 (+ 562us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:48:08.108530 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:48:08.108533 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:48:08.108925 (+ 392us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 14:48:08.109227 (+ 302us) server_negotiation.cc:1035] Waiting for connection context
0620 14:48:08.110011 (+ 784us) server_negotiation.cc:299] Negotiation successful
0620 14:48:08.110334 (+ 323us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":59}
I20260620 14:48:08.112293 7871 tablet_replica.cc:330] T a676dfab9a7c47b2860003164f56d54e P 76303ed5f6ac43b7866d8efd0313bdc2: stopping tablet replica
I20260620 14:48:08.112807 7871 raft_consensus.cc:2229] T a676dfab9a7c47b2860003164f56d54e P 76303ed5f6ac43b7866d8efd0313bdc2 [term 1 LEADER]: Raft consensus shutting down.
I20260620 14:48:08.113613 7871 raft_consensus.cc:2258] T a676dfab9a7c47b2860003164f56d54e P 76303ed5f6ac43b7866d8efd0313bdc2 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 14:48:08.116681 7871 ts_tablet_manager.cc:1905] T a676dfab9a7c47b2860003164f56d54e P 76303ed5f6ac43b7866d8efd0313bdc2: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 14:48:08.123267 7871 ts_tablet_manager.cc:1918] T a676dfab9a7c47b2860003164f56d54e P 76303ed5f6ac43b7866d8efd0313bdc2: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260620 14:48:08.123450 7871 log.cc:1196] T a676dfab9a7c47b2860003164f56d54e P 76303ed5f6ac43b7866d8efd0313bdc2: Deleting WAL directory at /tmp/dist-test-taskUci3DU/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781966829091300-388-0/minicluster-data/ts-0/wal/wals/a676dfab9a7c47b2860003164f56d54e
I20260620 14:48:08.124204 7871 ts_tablet_manager.cc:1939] T a676dfab9a7c47b2860003164f56d54e P 76303ed5f6ac43b7866d8efd0313bdc2: Deleting consensus metadata
I20260620 14:48:08.130170 7818 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:48:08.090286 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51932 (local address 127.0.97.60:37899)
0620 14:48:08.090468 (+ 182us) server_negotiation.cc:206] Beginning negotiation
0620 14:48:08.090475 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0620 14:48:08.112367 (+ 21892us) server_negotiation.cc:407] Connection header received
0620 14:48:08.112620 (+ 253us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:48:08.112630 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:48:08.112704 (+ 74us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:48:08.112822 (+ 118us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:48:08.114907 (+ 2085us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:48:08.115759 (+ 852us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:48:08.118443 (+ 2684us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:48:08.119112 (+ 669us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:48:08.119229 (+ 117us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:48:08.122986 (+ 3757us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 14:48:08.122993 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 14:48:08.122998 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 14:48:08.123044 (+ 46us) server_negotiation.cc:922] calling sasl_server_start()
0620 14:48:08.126882 (+ 3838us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:48:08.127802 (+ 920us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:48:08.127809 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:48:08.127813 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:48:08.127979 (+ 166us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:48:08.128489 (+ 510us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:48:08.128494 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:48:08.128498 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:48:08.128893 (+ 395us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 14:48:08.129138 (+ 245us) server_negotiation.cc:1035] Waiting for connection context
0620 14:48:08.129558 (+ 420us) server_negotiation.cc:299] Negotiation successful
0620 14:48:08.129794 (+ 236us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":47}
I20260620 14:48:08.137446 7872 ts_tablet_manager.cc:1918] T a676dfab9a7c47b2860003164f56d54e P 790f4a4136784937b4bfc2572c01a54b: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260620 14:48:08.137671 7872 log.cc:1196] T a676dfab9a7c47b2860003164f56d54e P 790f4a4136784937b4bfc2572c01a54b: Deleting WAL directory at /tmp/dist-test-taskUci3DU/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781966829091300-388-0/minicluster-data/ts-1/wal/wals/a676dfab9a7c47b2860003164f56d54e
I20260620 14:48:08.138341 7872 ts_tablet_manager.cc:1939] T a676dfab9a7c47b2860003164f56d54e P 790f4a4136784937b4bfc2572c01a54b: Deleting consensus metadata
I20260620 14:48:08.252480 6536 catalog_manager.cc:4841] TS 790f4a4136784937b4bfc2572c01a54b (127.0.97.2:36327): tablet a676dfab9a7c47b2860003164f56d54e (table test-table [id=b0513397257647398cf696e60f8c1af5]) successfully deleted
I20260620 14:48:08.253623 6534 catalog_manager.cc:4841] TS 76303ed5f6ac43b7866d8efd0313bdc2 (127.0.97.1:39249): tablet a676dfab9a7c47b2860003164f56d54e (table test-table [id=b0513397257647398cf696e60f8c1af5]) successfully deleted
I20260620 14:48:08.254281 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskUci3DU/build/asan/bin/kudu with pid 6518
I20260620 14:48:08.300239 7818 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:48:08.281228 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51946 (local address 127.0.97.60:37899)
0620 14:48:08.281456 (+ 228us) server_negotiation.cc:206] Beginning negotiation
0620 14:48:08.281466 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0620 14:48:08.282548 (+ 1082us) server_negotiation.cc:407] Connection header received
0620 14:48:08.282851 (+ 303us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:48:08.282857 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:48:08.282934 (+ 77us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:48:08.282997 (+ 63us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:48:08.285731 (+ 2734us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:48:08.286864 (+ 1133us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:48:08.289409 (+ 2545us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:48:08.289843 (+ 434us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:48:08.290004 (+ 161us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:48:08.294037 (+ 4033us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 14:48:08.294045 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 14:48:08.294050 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 14:48:08.294106 (+ 56us) server_negotiation.cc:922] calling sasl_server_start()
0620 14:48:08.297215 (+ 3109us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:48:08.297947 (+ 732us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:48:08.297952 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:48:08.297954 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:48:08.298075 (+ 121us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:48:08.298473 (+ 398us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:48:08.298477 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:48:08.298479 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:48:08.298890 (+ 411us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 14:48:08.299163 (+ 273us) server_negotiation.cc:1035] Waiting for connection context
0620 14:48:08.299533 (+ 370us) server_negotiation.cc:299] Negotiation successful
0620 14:48:08.299877 (+ 344us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":51}
I20260620 14:48:08.301734 7820 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:48:08.281065 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48096 (local address 127.0.97.61:41719)
0620 14:48:08.281267 (+ 202us) server_negotiation.cc:206] Beginning negotiation
0620 14:48:08.281274 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0620 14:48:08.282439 (+ 1165us) server_negotiation.cc:407] Connection header received
0620 14:48:08.282748 (+ 309us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:48:08.282756 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:48:08.282847 (+ 91us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:48:08.282943 (+ 96us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:48:08.285733 (+ 2790us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:48:08.286839 (+ 1106us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:48:08.289210 (+ 2371us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:48:08.289920 (+ 710us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:48:08.290057 (+ 137us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:48:08.294901 (+ 4844us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 14:48:08.294908 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 14:48:08.294913 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 14:48:08.294956 (+ 43us) server_negotiation.cc:922] calling sasl_server_start()
0620 14:48:08.298419 (+ 3463us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:48:08.299196 (+ 777us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:48:08.299201 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:48:08.299205 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:48:08.299327 (+ 122us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:48:08.299849 (+ 522us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:48:08.299853 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:48:08.299857 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:48:08.300246 (+ 389us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 14:48:08.300489 (+ 243us) server_negotiation.cc:1035] Waiting for connection context
0620 14:48:08.301103 (+ 614us) server_negotiation.cc:299] Negotiation successful
0620 14:48:08.301409 (+ 306us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":66}
I20260620 14:48:09.051192 7882 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:48:09.050340 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:39515 (local address 127.0.97.2:35743)
0620 14:48:09.050851 (+ 511us) negotiation.cc:107] Waiting for socket to connect
0620 14:48:09.050956 (+ 105us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:39515: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":372,"thread_start_us":273,"threads_started":1}
W20260620 14:48:09.051716 7674 heartbeater.cc:644] Failed to heartbeat to 127.0.97.62:39515 (0 consecutive failures): Network error: Failed to send heartbeat to master: Client connection negotiation failed: client connection to 127.0.97.62:39515: connect: Connection refused (error 111)
I20260620 14:48:09.054358 7883 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:48:09.053433 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:39515 (local address 127.0.97.3:34023)
0620 14:48:09.053976 (+ 543us) negotiation.cc:107] Waiting for socket to connect
0620 14:48:09.054130 (+ 154us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:39515: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":412,"thread_start_us":321,"threads_started":1}
W20260620 14:48:09.054781 7814 heartbeater.cc:644] Failed to heartbeat to 127.0.97.62:39515 (0 consecutive failures): Network error: Failed to send heartbeat to master: Client connection negotiation failed: client connection to 127.0.97.62:39515: connect: Connection refused (error 111)
I20260620 14:48:09.057199 7884 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:48:09.056501 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:39515 (local address 127.0.97.1:38497)
0620 14:48:09.056934 (+ 433us) negotiation.cc:107] Waiting for socket to connect
0620 14:48:09.057038 (+ 104us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:39515: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":336,"thread_start_us":238,"threads_started":1}
W20260620 14:48:09.057632 7526 heartbeater.cc:644] Failed to heartbeat to 127.0.97.62:39515 (0 consecutive failures): Network error: Failed to send heartbeat to master: Client connection negotiation failed: client connection to 127.0.97.62:39515: connect: Connection refused (error 111)
I20260620 14:48:09.572427 7885 raft_consensus.cc:482] T 00000000000000000000000000000000 P 55c0ce5da3694b759b5b3113647c3eaa [term 1 FOLLOWER]: Starting pre-election (detected failure of leader d663f8b7f54d4fe68ce4af5b87767fed)
I20260620 14:48:09.572664 7885 raft_consensus.cc:504] T 00000000000000000000000000000000 P 55c0ce5da3694b759b5b3113647c3eaa [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "d663f8b7f54d4fe68ce4af5b87767fed" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 39515 } } peers { permanent_uuid: "55c0ce5da3694b759b5b3113647c3eaa" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 41719 } attrs { promote: false } } peers { permanent_uuid: "e4dec65330ba4aefa643b5c1fcbc31ae" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 37899 } attrs { promote: false } }
I20260620 14:48:09.573622 7885 leader_election.cc:288] T 00000000000000000000000000000000 P 55c0ce5da3694b759b5b3113647c3eaa [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers d663f8b7f54d4fe68ce4af5b87767fed (127.0.97.62:39515), e4dec65330ba4aefa643b5c1fcbc31ae (127.0.97.60:37899)
I20260620 14:48:09.574879 7886 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:48:09.573834 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:39515 (local address 127.0.0.1:43104)
0620 14:48:09.574454 (+ 620us) negotiation.cc:107] Waiting for socket to connect
0620 14:48:09.574666 (+ 212us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:39515: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":465,"thread_start_us":273,"threads_started":1}
W20260620 14:48:09.575265 7134 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.97.62:39515: connect: Connection refused (error 111)
I20260620 14:48:09.576967 7886 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:48:09.576580 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:39515 (local address 127.0.0.1:43110)
0620 14:48:09.576743 (+ 163us) negotiation.cc:107] Waiting for socket to connect
0620 14:48:09.576827 (+ 84us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:39515: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":61}
W20260620 14:48:09.577159 7134 leader_election.cc:334] T 00000000000000000000000000000000 P 55c0ce5da3694b759b5b3113647c3eaa [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer d663f8b7f54d4fe68ce4af5b87767fed (127.0.97.62:39515): Network error: Client connection negotiation failed: client connection to 127.0.97.62:39515: connect: Connection refused (error 111)
I20260620 14:48:09.580286 7890 raft_consensus.cc:482] T 00000000000000000000000000000000 P e4dec65330ba4aefa643b5c1fcbc31ae [term 1 FOLLOWER]: Starting pre-election (detected failure of leader d663f8b7f54d4fe68ce4af5b87767fed)
I20260620 14:48:09.580621 7887 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:48:09.573859 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.60:37899 (local address 127.0.0.1:41748)
0620 14:48:09.574544 (+ 685us) negotiation.cc:107] Waiting for socket to connect
0620 14:48:09.574569 (+ 25us) client_negotiation.cc:174] Beginning negotiation
0620 14:48:09.574762 (+ 193us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 14:48:09.575559 (+ 797us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 14:48:09.575564 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 14:48:09.575590 (+ 26us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 14:48:09.575995 (+ 405us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:48:09.576005 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:48:09.577332 (+ 1327us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:48:09.577339 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:48:09.578692 (+ 1353us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:48:09.578706 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:48:09.579830 (+ 1124us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:48:09.579836 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:48:09.580003 (+ 167us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:48:09.580027 (+ 24us) client_negotiation.cc:769] Sending connection context
0620 14:48:09.580140 (+ 113us) client_negotiation.cc:240] Negotiation successful
0620 14:48:09.580280 (+ 140us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":527,"thread_start_us":357,"threads_started":1}
I20260620 14:48:09.580638 7888 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:48:09.574101 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41748 (local address 127.0.97.60:37899)
0620 14:48:09.575264 (+ 1163us) server_negotiation.cc:206] Beginning negotiation
0620 14:48:09.575269 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 14:48:09.575284 (+ 15us) server_negotiation.cc:407] Connection header received
0620 14:48:09.575344 (+ 60us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:48:09.575348 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:48:09.575410 (+ 62us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:48:09.575539 (+ 129us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 14:48:09.576250 (+ 711us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:48:09.577194 (+ 944us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:48:09.578845 (+ 1651us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:48:09.579666 (+ 821us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:48:09.579811 (+ 145us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:48:09.579904 (+ 93us) server_negotiation.cc:1035] Waiting for connection context
0620 14:48:09.580167 (+ 263us) server_negotiation.cc:299] Negotiation successful
0620 14:48:09.580300 (+ 133us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1069,"thread_start_us":933,"threads_started":1}
I20260620 14:48:09.580569 7890 raft_consensus.cc:504] T 00000000000000000000000000000000 P e4dec65330ba4aefa643b5c1fcbc31ae [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "d663f8b7f54d4fe68ce4af5b87767fed" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 39515 } } peers { permanent_uuid: "55c0ce5da3694b759b5b3113647c3eaa" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 41719 } attrs { promote: false } } peers { permanent_uuid: "e4dec65330ba4aefa643b5c1fcbc31ae" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 37899 } attrs { promote: false } }
I20260620 14:48:09.581351 7358 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "55c0ce5da3694b759b5b3113647c3eaa" candidate_term: 2 candidate_status { last_received { term: 1 index: 12 } } ignore_live_leader: false dest_uuid: "e4dec65330ba4aefa643b5c1fcbc31ae" is_pre_election: true
I20260620 14:48:09.581748 7358 raft_consensus.cc:2454] T 00000000000000000000000000000000 P e4dec65330ba4aefa643b5c1fcbc31ae [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 55c0ce5da3694b759b5b3113647c3eaa in term 1.
I20260620 14:48:09.582239 7135 leader_election.cc:302] T 00000000000000000000000000000000 P 55c0ce5da3694b759b5b3113647c3eaa [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 55c0ce5da3694b759b5b3113647c3eaa, e4dec65330ba4aefa643b5c1fcbc31ae; no voters: d663f8b7f54d4fe68ce4af5b87767fed
I20260620 14:48:09.582681 7885 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 55c0ce5da3694b759b5b3113647c3eaa [term 1 FOLLOWER]: Leader pre-election won for term 2
I20260620 14:48:09.582815 7885 raft_consensus.cc:482] T 00000000000000000000000000000000 P 55c0ce5da3694b759b5b3113647c3eaa [term 1 FOLLOWER]: Starting leader election (detected failure of leader d663f8b7f54d4fe68ce4af5b87767fed)
I20260620 14:48:09.582907 7885 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 55c0ce5da3694b759b5b3113647c3eaa [term 1 FOLLOWER]: Advancing to term 2
I20260620 14:48:09.584307 7890 leader_election.cc:288] T 00000000000000000000000000000000 P e4dec65330ba4aefa643b5c1fcbc31ae [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers d663f8b7f54d4fe68ce4af5b87767fed (127.0.97.62:39515), 55c0ce5da3694b759b5b3113647c3eaa (127.0.97.61:41719)
I20260620 14:48:09.586004 7892 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:48:09.582624 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:39515 (local address 127.0.0.1:43128)
0620 14:48:09.585409 (+ 2785us) negotiation.cc:107] Waiting for socket to connect
0620 14:48:09.585655 (+ 246us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:39515: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":2662,"thread_start_us":2931,"threads_started":1}
I20260620 14:48:09.586205 7885 raft_consensus.cc:504] T 00000000000000000000000000000000 P 55c0ce5da3694b759b5b3113647c3eaa [term 2 FOLLOWER]: Starting leader election with config: opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "d663f8b7f54d4fe68ce4af5b87767fed" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 39515 } } peers { permanent_uuid: "55c0ce5da3694b759b5b3113647c3eaa" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 41719 } attrs { promote: false } } peers { permanent_uuid: "e4dec65330ba4aefa643b5c1fcbc31ae" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 37899 } attrs { promote: false } }
W20260620 14:48:09.586375 7331 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.97.62:39515: connect: Connection refused (error 111)
I20260620 14:48:09.587035 7885 leader_election.cc:288] T 00000000000000000000000000000000 P 55c0ce5da3694b759b5b3113647c3eaa [CANDIDATE]: Term 2 election: Requested vote from peers d663f8b7f54d4fe68ce4af5b87767fed (127.0.97.62:39515), e4dec65330ba4aefa643b5c1fcbc31ae (127.0.97.60:37899)
I20260620 14:48:09.587944 7887 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:48:09.587153 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:39515 (local address 127.0.0.1:43140)
0620 14:48:09.587658 (+ 505us) negotiation.cc:107] Waiting for socket to connect
0620 14:48:09.587787 (+ 129us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:39515: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":333}
I20260620 14:48:09.588260 7358 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "55c0ce5da3694b759b5b3113647c3eaa" candidate_term: 2 candidate_status { last_received { term: 1 index: 12 } } ignore_live_leader: false dest_uuid: "e4dec65330ba4aefa643b5c1fcbc31ae"
I20260620 14:48:09.588531 7358 raft_consensus.cc:3037] T 00000000000000000000000000000000 P e4dec65330ba4aefa643b5c1fcbc31ae [term 1 FOLLOWER]: Advancing to term 2
I20260620 14:48:09.589040 7887 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:48:09.588650 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:39515 (local address 127.0.0.1:43154)
0620 14:48:09.588814 (+ 164us) negotiation.cc:107] Waiting for socket to connect
0620 14:48:09.588896 (+ 82us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:39515: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":54}
W20260620 14:48:09.589210 7134 leader_election.cc:334] T 00000000000000000000000000000000 P 55c0ce5da3694b759b5b3113647c3eaa [CANDIDATE]: Term 2 election: RPC error from VoteRequest() call to peer d663f8b7f54d4fe68ce4af5b87767fed (127.0.97.62:39515): Network error: Client connection negotiation failed: client connection to 127.0.97.62:39515: connect: Connection refused (error 111)
I20260620 14:48:09.589747 7892 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:48:09.589003 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:39515 (local address 127.0.0.1:43166)
0620 14:48:09.589496 (+ 493us) negotiation.cc:107] Waiting for socket to connect
0620 14:48:09.589580 (+ 84us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:39515: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":366}
W20260620 14:48:09.589941 7331 leader_election.cc:334] T 00000000000000000000000000000000 P e4dec65330ba4aefa643b5c1fcbc31ae [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer d663f8b7f54d4fe68ce4af5b87767fed (127.0.97.62:39515): Network error: Client connection negotiation failed: client connection to 127.0.97.62:39515: connect: Connection refused (error 111)
I20260620 14:48:09.591308 7358 raft_consensus.cc:2454] T 00000000000000000000000000000000 P e4dec65330ba4aefa643b5c1fcbc31ae [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 55c0ce5da3694b759b5b3113647c3eaa in term 2.
I20260620 14:48:09.591773 7135 leader_election.cc:302] T 00000000000000000000000000000000 P 55c0ce5da3694b759b5b3113647c3eaa [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 55c0ce5da3694b759b5b3113647c3eaa, e4dec65330ba4aefa643b5c1fcbc31ae; no voters: d663f8b7f54d4fe68ce4af5b87767fed
I20260620 14:48:09.592067 7885 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 55c0ce5da3694b759b5b3113647c3eaa [term 2 FOLLOWER]: Leader election won for term 2
I20260620 14:48:09.592639 7891 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:48:09.582896 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.61:41719 (local address 127.0.0.1:48572)
0620 14:48:09.585879 (+ 2983us) negotiation.cc:107] Waiting for socket to connect
0620 14:48:09.585903 (+ 24us) client_negotiation.cc:174] Beginning negotiation
0620 14:48:09.586048 (+ 145us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 14:48:09.586438 (+ 390us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 14:48:09.586442 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 14:48:09.586483 (+ 41us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 14:48:09.586853 (+ 370us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:48:09.586867 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:48:09.588564 (+ 1697us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:48:09.588567 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:48:09.590563 (+ 1996us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:48:09.590578 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:48:09.591828 (+ 1250us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:48:09.591833 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:48:09.592025 (+ 192us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:48:09.592052 (+ 27us) client_negotiation.cc:769] Sending connection context
0620 14:48:09.592162 (+ 110us) client_negotiation.cc:240] Negotiation successful
0620 14:48:09.592297 (+ 135us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":2822,"thread_start_us":2241,"threads_started":1}
I20260620 14:48:09.592667 7893 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:48:09.583258 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48572 (local address 127.0.97.61:41719)
0620 14:48:09.585003 (+ 1745us) server_negotiation.cc:206] Beginning negotiation
0620 14:48:09.585009 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 14:48:09.586159 (+ 1150us) server_negotiation.cc:407] Connection header received
0620 14:48:09.586226 (+ 67us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:48:09.586231 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:48:09.586294 (+ 63us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:48:09.586401 (+ 107us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 14:48:09.587094 (+ 693us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:48:09.588379 (+ 1285us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:48:09.590742 (+ 2363us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:48:09.591667 (+ 925us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:48:09.591815 (+ 148us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:48:09.591936 (+ 121us) server_negotiation.cc:1035] Waiting for connection context
0620 14:48:09.592197 (+ 261us) server_negotiation.cc:299] Negotiation successful
0620 14:48:09.592333 (+ 136us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1594,"thread_start_us":1353,"threads_started":1}
I20260620 14:48:09.592974 7885 raft_consensus.cc:686] T 00000000000000000000000000000000 P 55c0ce5da3694b759b5b3113647c3eaa [term 2 LEADER]: Becoming Leader. State: Replica: 55c0ce5da3694b759b5b3113647c3eaa, State: Running, Role: LEADER
I20260620 14:48:09.593402 7158 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "e4dec65330ba4aefa643b5c1fcbc31ae" candidate_term: 2 candidate_status { last_received { term: 1 index: 12 } } ignore_live_leader: false dest_uuid: "55c0ce5da3694b759b5b3113647c3eaa" is_pre_election: true
I20260620 14:48:09.593465 7885 consensus_queue.cc:237] T 00000000000000000000000000000000 P 55c0ce5da3694b759b5b3113647c3eaa [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 12, Committed index: 12, Last appended: 1.12, Last appended by leader: 12, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "d663f8b7f54d4fe68ce4af5b87767fed" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 39515 } } peers { permanent_uuid: "55c0ce5da3694b759b5b3113647c3eaa" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 41719 } attrs { promote: false } } peers { permanent_uuid: "e4dec65330ba4aefa643b5c1fcbc31ae" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 37899 } attrs { promote: false } }
I20260620 14:48:09.594139 7333 leader_election.cc:302] T 00000000000000000000000000000000 P e4dec65330ba4aefa643b5c1fcbc31ae [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: e4dec65330ba4aefa643b5c1fcbc31ae; no voters: 55c0ce5da3694b759b5b3113647c3eaa, d663f8b7f54d4fe68ce4af5b87767fed
I20260620 14:48:09.594491 7890 raft_consensus.cc:2726] T 00000000000000000000000000000000 P e4dec65330ba4aefa643b5c1fcbc31ae [term 2 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260620 14:48:09.595392 7895 sys_catalog.cc:455] T 00000000000000000000000000000000 P 55c0ce5da3694b759b5b3113647c3eaa [sys.catalog]: SysCatalogTable state changed. Reason: New leader 55c0ce5da3694b759b5b3113647c3eaa. Latest consensus state: current_term: 2 leader_uuid: "55c0ce5da3694b759b5b3113647c3eaa" committed_config { opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "d663f8b7f54d4fe68ce4af5b87767fed" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 39515 } } peers { permanent_uuid: "55c0ce5da3694b759b5b3113647c3eaa" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 41719 } attrs { promote: false } } peers { permanent_uuid: "e4dec65330ba4aefa643b5c1fcbc31ae" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 37899 } attrs { promote: false } } }
I20260620 14:48:09.595826 7895 sys_catalog.cc:458] T 00000000000000000000000000000000 P 55c0ce5da3694b759b5b3113647c3eaa [sys.catalog]: This master's current role is: LEADER
I20260620 14:48:09.596477 7897 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 14:48:09.599982 7897 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 14:48:09.600826 7897 catalog_manager.cc:1259] Loaded cluster ID: a48bc503bb2347ef82ed8e77d4e21c70
I20260620 14:48:09.600930 7897 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 14:48:09.602031 7897 catalog_manager.cc:1504] Loading token signing keys...
I20260620 14:48:09.602851 7897 catalog_manager.cc:5879] T 00000000000000000000000000000000 P 55c0ce5da3694b759b5b3113647c3eaa: Loaded TSK: 0
I20260620 14:48:09.603749 7897 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260620 14:48:09.641804 7147 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:48096:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260620 14:48:09.649361 7147 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260620 14:48:09.654168 7887 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:48:09.653676 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:39515 (local address 127.0.0.1:43184)
0620 14:48:09.653867 (+ 191us) negotiation.cc:107] Waiting for socket to connect
0620 14:48:09.653972 (+ 105us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:39515: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":58}
I20260620 14:48:09.654207 7358 raft_consensus.cc:1261] T 00000000000000000000000000000000 P e4dec65330ba4aefa643b5c1fcbc31ae [term 2 FOLLOWER]: Refusing update from remote peer 55c0ce5da3694b759b5b3113647c3eaa: Log matching property violated. Preceding OpId in replica: term: 1 index: 12. Preceding OpId from leader: term: 2 index: 14. (index mismatch)
I20260620 14:48:09.654958 7895 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 55c0ce5da3694b759b5b3113647c3eaa [LEADER]: Connected to new peer: Peer: permanent_uuid: "e4dec65330ba4aefa643b5c1fcbc31ae" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 37899 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 13, Last known committed idx: 12, Time since last communication: 0.000s
I20260620 14:48:09.655324 7887 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:48:09.654906 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:39515 (local address 127.0.0.1:43200)
0620 14:48:09.655064 (+ 158us) negotiation.cc:107] Waiting for socket to connect
0620 14:48:09.655135 (+ 71us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:39515: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":53}
W20260620 14:48:09.655539 7134 consensus_peers.cc:487] T 00000000000000000000000000000000 P 55c0ce5da3694b759b5b3113647c3eaa -> Peer d663f8b7f54d4fe68ce4af5b87767fed (127.0.97.62:39515): Couldn't send request to peer d663f8b7f54d4fe68ce4af5b87767fed. Status: Network error: Client connection negotiation failed: client connection to 127.0.97.62:39515: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20260620 14:48:09.657634 7890 sys_catalog.cc:455] T 00000000000000000000000000000000 P e4dec65330ba4aefa643b5c1fcbc31ae [sys.catalog]: SysCatalogTable state changed. Reason: New leader 55c0ce5da3694b759b5b3113647c3eaa. Latest consensus state: current_term: 2 leader_uuid: "55c0ce5da3694b759b5b3113647c3eaa" committed_config { opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "d663f8b7f54d4fe68ce4af5b87767fed" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 39515 } } peers { permanent_uuid: "55c0ce5da3694b759b5b3113647c3eaa" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 41719 } attrs { promote: false } } peers { permanent_uuid: "e4dec65330ba4aefa643b5c1fcbc31ae" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 37899 } attrs { promote: false } } }
I20260620 14:48:09.657964 7890 sys_catalog.cc:458] T 00000000000000000000000000000000 P e4dec65330ba4aefa643b5c1fcbc31ae [sys.catalog]: This master's current role is: FOLLOWER
I20260620 14:48:09.659584 7895 sys_catalog.cc:455] T 00000000000000000000000000000000 P 55c0ce5da3694b759b5b3113647c3eaa [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 2 leader_uuid: "55c0ce5da3694b759b5b3113647c3eaa" committed_config { opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "d663f8b7f54d4fe68ce4af5b87767fed" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 39515 } } peers { permanent_uuid: "55c0ce5da3694b759b5b3113647c3eaa" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 41719 } attrs { promote: false } } peers { permanent_uuid: "e4dec65330ba4aefa643b5c1fcbc31ae" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 37899 } attrs { promote: false } } }
I20260620 14:48:09.659942 7895 sys_catalog.cc:458] T 00000000000000000000000000000000 P 55c0ce5da3694b759b5b3113647c3eaa [sys.catalog]: This master's current role is: LEADER
I20260620 14:48:09.660528 7890 sys_catalog.cc:455] T 00000000000000000000000000000000 P e4dec65330ba4aefa643b5c1fcbc31ae [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 2 leader_uuid: "55c0ce5da3694b759b5b3113647c3eaa" committed_config { opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "d663f8b7f54d4fe68ce4af5b87767fed" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 39515 } } peers { permanent_uuid: "55c0ce5da3694b759b5b3113647c3eaa" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 41719 } attrs { promote: false } } peers { permanent_uuid: "e4dec65330ba4aefa643b5c1fcbc31ae" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 37899 } attrs { promote: false } } }
I20260620 14:48:09.660780 7890 sys_catalog.cc:458] T 00000000000000000000000000000000 P e4dec65330ba4aefa643b5c1fcbc31ae [sys.catalog]: This master's current role is: FOLLOWER
I20260620 14:48:09.676244 7886 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:48:09.669540 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.2:36327 (local address 127.0.0.1:40074)
0620 14:48:09.669841 (+ 301us) negotiation.cc:107] Waiting for socket to connect
0620 14:48:09.669861 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0620 14:48:09.669985 (+ 124us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 14:48:09.670946 (+ 961us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 14:48:09.670951 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 14:48:09.670974 (+ 23us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 14:48:09.671220 (+ 246us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:48:09.671228 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:48:09.672529 (+ 1301us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:48:09.672534 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:48:09.674074 (+ 1540us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:48:09.674088 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:48:09.675444 (+ 1356us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:48:09.675451 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:48:09.675681 (+ 230us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:48:09.675720 (+ 39us) client_negotiation.cc:769] Sending connection context
0620 14:48:09.675788 (+ 68us) client_negotiation.cc:240] Negotiation successful
0620 14:48:09.675934 (+ 146us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":176}
I20260620 14:48:09.677217 7907 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:48:09.670242 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40074 (local address 127.0.97.2:36327)
0620 14:48:09.670683 (+ 441us) server_negotiation.cc:206] Beginning negotiation
0620 14:48:09.670689 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 14:48:09.670705 (+ 16us) server_negotiation.cc:407] Connection header received
0620 14:48:09.670771 (+ 66us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:48:09.670775 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:48:09.670824 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:48:09.670931 (+ 107us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 14:48:09.671364 (+ 433us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:48:09.672385 (+ 1021us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:48:09.674267 (+ 1882us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:48:09.675193 (+ 926us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:48:09.676703 (+ 1510us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:48:09.676809 (+ 106us) server_negotiation.cc:1035] Waiting for connection context
0620 14:48:09.676851 (+ 42us) server_negotiation.cc:299] Negotiation successful
0620 14:48:09.676952 (+ 101us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":318,"thread_start_us":220,"threads_started":1}
I20260620 14:48:09.678061 7608 tablet_service.cc:1467] Processing CreateTablet for tablet 77de1104f698409ab9e3664c1465711f (DEFAULT_TABLE table=test-table [id=27bc53d06b394ae5a47a19167b6830e1]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 14:48:09.678525 7608 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 77de1104f698409ab9e3664c1465711f. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 14:48:09.685052 7908 tablet_bootstrap.cc:492] T 77de1104f698409ab9e3664c1465711f P 790f4a4136784937b4bfc2572c01a54b: Bootstrap starting.
I20260620 14:48:09.686548 7887 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:48:09.668312 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:39249 (local address 127.0.0.1:52102)
0620 14:48:09.668500 (+ 188us) negotiation.cc:107] Waiting for socket to connect
0620 14:48:09.668521 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0620 14:48:09.668638 (+ 117us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 14:48:09.679552 (+ 10914us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 14:48:09.679559 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 14:48:09.679616 (+ 57us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 14:48:09.679800 (+ 184us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:48:09.679811 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:48:09.682072 (+ 2261us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:48:09.682078 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:48:09.684145 (+ 2067us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:48:09.684161 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:48:09.685732 (+ 1571us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:48:09.685739 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:48:09.685965 (+ 226us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:48:09.686035 (+ 70us) client_negotiation.cc:769] Sending connection context
0620 14:48:09.686109 (+ 74us) client_negotiation.cc:240] Negotiation successful
0620 14:48:09.686267 (+ 158us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":57,"mutex_wait_us":12}
I20260620 14:48:09.686797 7906 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:48:09.669988 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41754 (local address 127.0.97.3:46083)
0620 14:48:09.680030 (+ 10042us) server_negotiation.cc:206] Beginning negotiation
0620 14:48:09.680038 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0620 14:48:09.680061 (+ 23us) server_negotiation.cc:407] Connection header received
0620 14:48:09.680151 (+ 90us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:48:09.680156 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:48:09.680251 (+ 95us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:48:09.680522 (+ 271us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 14:48:09.681188 (+ 666us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:48:09.682214 (+ 1026us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:48:09.684117 (+ 1903us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:48:09.685536 (+ 1419us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:48:09.685694 (+ 158us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:48:09.685816 (+ 122us) server_negotiation.cc:1035] Waiting for connection context
0620 14:48:09.686369 (+ 553us) server_negotiation.cc:299] Negotiation successful
0620 14:48:09.686510 (+ 141us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":9923,"thread_start_us":9832,"threads_started":1}
I20260620 14:48:09.687186 7905 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:48:09.669897 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.3:46083 (local address 127.0.0.1:41754)
0620 14:48:09.670460 (+ 563us) negotiation.cc:107] Waiting for socket to connect
0620 14:48:09.670479 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0620 14:48:09.670585 (+ 106us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 14:48:09.680544 (+ 9959us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 14:48:09.680549 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 14:48:09.680577 (+ 28us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 14:48:09.680990 (+ 413us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:48:09.681003 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:48:09.682348 (+ 1345us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:48:09.682355 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:48:09.683934 (+ 1579us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:48:09.683948 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:48:09.685775 (+ 1827us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:48:09.685781 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:48:09.685999 (+ 218us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:48:09.686036 (+ 37us) client_negotiation.cc:769] Sending connection context
0620 14:48:09.686132 (+ 96us) client_negotiation.cc:240] Negotiation successful
0620 14:48:09.686275 (+ 143us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":413,"mutex_wait_us":25,"thread_start_us":253,"threads_started":1}
I20260620 14:48:09.687985 7904 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:48:09.668286 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52102 (local address 127.0.97.1:39249)
0620 14:48:09.679082 (+ 10796us) server_negotiation.cc:206] Beginning negotiation
0620 14:48:09.679092 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0620 14:48:09.679118 (+ 26us) server_negotiation.cc:407] Connection header received
0620 14:48:09.679222 (+ 104us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:48:09.679230 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:48:09.679335 (+ 105us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:48:09.679490 (+ 155us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 14:48:09.680742 (+ 1252us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:48:09.681905 (+ 1163us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:48:09.684321 (+ 2416us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:48:09.685536 (+ 1215us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:48:09.687449 (+ 1913us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:48:09.687561 (+ 112us) server_negotiation.cc:1035] Waiting for connection context
0620 14:48:09.687639 (+ 78us) server_negotiation.cc:299] Negotiation successful
0620 14:48:09.687781 (+ 142us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":10616,"thread_start_us":12058,"threads_started":1}
I20260620 14:48:09.688081 7908 tablet_bootstrap.cc:654] T 77de1104f698409ab9e3664c1465711f P 790f4a4136784937b4bfc2572c01a54b: Neither blocks nor log segments found. Creating new log.
I20260620 14:48:09.688033 7748 tablet_service.cc:1467] Processing CreateTablet for tablet 77de1104f698409ab9e3664c1465711f (DEFAULT_TABLE table=test-table [id=27bc53d06b394ae5a47a19167b6830e1]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 14:48:09.688503 7748 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 77de1104f698409ab9e3664c1465711f. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 14:48:09.688670 7460 tablet_service.cc:1467] Processing CreateTablet for tablet 77de1104f698409ab9e3664c1465711f (DEFAULT_TABLE table=test-table [id=27bc53d06b394ae5a47a19167b6830e1]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 14:48:09.689044 7460 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 77de1104f698409ab9e3664c1465711f. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 14:48:09.690361 7908 tablet_bootstrap.cc:492] T 77de1104f698409ab9e3664c1465711f P 790f4a4136784937b4bfc2572c01a54b: No bootstrap required, opened a new log
I20260620 14:48:09.690498 7908 ts_tablet_manager.cc:1397] T 77de1104f698409ab9e3664c1465711f P 790f4a4136784937b4bfc2572c01a54b: Time spent bootstrapping tablet: real 0.006s user 0.003s sys 0.000s
I20260620 14:48:09.691270 7908 raft_consensus.cc:348] T 77de1104f698409ab9e3664c1465711f P 790f4a4136784937b4bfc2572c01a54b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "76303ed5f6ac43b7866d8efd0313bdc2" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 39249 } } peers { permanent_uuid: "790f4a4136784937b4bfc2572c01a54b" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 36327 } } peers { permanent_uuid: "361b2f75b32a48cda1d69fd295208d8d" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 46083 } }
I20260620 14:48:09.691529 7908 raft_consensus.cc:374] T 77de1104f698409ab9e3664c1465711f P 790f4a4136784937b4bfc2572c01a54b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 14:48:09.691646 7908 raft_consensus.cc:729] T 77de1104f698409ab9e3664c1465711f P 790f4a4136784937b4bfc2572c01a54b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 790f4a4136784937b4bfc2572c01a54b, State: Initialized, Role: FOLLOWER
I20260620 14:48:09.691941 7908 consensus_queue.cc:260] T 77de1104f698409ab9e3664c1465711f P 790f4a4136784937b4bfc2572c01a54b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "76303ed5f6ac43b7866d8efd0313bdc2" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 39249 } } peers { permanent_uuid: "790f4a4136784937b4bfc2572c01a54b" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 36327 } } peers { permanent_uuid: "361b2f75b32a48cda1d69fd295208d8d" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 46083 } }
I20260620 14:48:09.694370 7908 ts_tablet_manager.cc:1428] T 77de1104f698409ab9e3664c1465711f P 790f4a4136784937b4bfc2572c01a54b: Time spent starting tablet: real 0.004s user 0.003s sys 0.000s
I20260620 14:48:09.697006 7675 heartbeater.cc:503] Master 127.0.97.61:41719 was elected leader, sending a full tablet report...
I20260620 14:48:09.699273 7913 tablet_bootstrap.cc:492] T 77de1104f698409ab9e3664c1465711f P 76303ed5f6ac43b7866d8efd0313bdc2: Bootstrap starting.
I20260620 14:48:09.701795 7913 tablet_bootstrap.cc:654] T 77de1104f698409ab9e3664c1465711f P 76303ed5f6ac43b7866d8efd0313bdc2: Neither blocks nor log segments found. Creating new log.
I20260620 14:48:09.704471 7911 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:48:09.693071 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:39515 (local address 127.0.97.2:43707)
0620 14:48:09.704129 (+ 11058us) negotiation.cc:107] Waiting for socket to connect
0620 14:48:09.704255 (+ 126us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:39515: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":10928,"thread_start_us":10754,"threads_started":1}
I20260620 14:48:09.705535 7911 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:48:09.705230 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:39515 (local address 127.0.97.2:50049)
0620 14:48:09.705361 (+ 131us) negotiation.cc:107] Waiting for socket to connect
0620 14:48:09.705435 (+ 74us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:39515: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":35}
I20260620 14:48:09.705704 7912 tablet_bootstrap.cc:492] T 77de1104f698409ab9e3664c1465711f P 361b2f75b32a48cda1d69fd295208d8d: Bootstrap starting.
I20260620 14:48:09.707027 7912 tablet_bootstrap.cc:654] T 77de1104f698409ab9e3664c1465711f P 361b2f75b32a48cda1d69fd295208d8d: Neither blocks nor log segments found. Creating new log.
I20260620 14:48:09.708910 7912 tablet_bootstrap.cc:492] T 77de1104f698409ab9e3664c1465711f P 361b2f75b32a48cda1d69fd295208d8d: No bootstrap required, opened a new log
I20260620 14:48:09.709057 7912 ts_tablet_manager.cc:1397] T 77de1104f698409ab9e3664c1465711f P 361b2f75b32a48cda1d69fd295208d8d: Time spent bootstrapping tablet: real 0.004s user 0.002s sys 0.000s
I20260620 14:48:09.709678 7912 raft_consensus.cc:348] T 77de1104f698409ab9e3664c1465711f P 361b2f75b32a48cda1d69fd295208d8d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "76303ed5f6ac43b7866d8efd0313bdc2" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 39249 } } peers { permanent_uuid: "790f4a4136784937b4bfc2572c01a54b" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 36327 } } peers { permanent_uuid: "361b2f75b32a48cda1d69fd295208d8d" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 46083 } }
I20260620 14:48:09.709864 7912 raft_consensus.cc:374] T 77de1104f698409ab9e3664c1465711f P 361b2f75b32a48cda1d69fd295208d8d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 14:48:09.709964 7912 raft_consensus.cc:729] T 77de1104f698409ab9e3664c1465711f P 361b2f75b32a48cda1d69fd295208d8d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 361b2f75b32a48cda1d69fd295208d8d, State: Initialized, Role: FOLLOWER
I20260620 14:48:09.710278 7913 tablet_bootstrap.cc:492] T 77de1104f698409ab9e3664c1465711f P 76303ed5f6ac43b7866d8efd0313bdc2: No bootstrap required, opened a new log
I20260620 14:48:09.710199 7912 consensus_queue.cc:260] T 77de1104f698409ab9e3664c1465711f P 361b2f75b32a48cda1d69fd295208d8d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "76303ed5f6ac43b7866d8efd0313bdc2" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 39249 } } peers { permanent_uuid: "790f4a4136784937b4bfc2572c01a54b" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 36327 } } peers { permanent_uuid: "361b2f75b32a48cda1d69fd295208d8d" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 46083 } }
I20260620 14:48:09.710436 7913 ts_tablet_manager.cc:1397] T 77de1104f698409ab9e3664c1465711f P 76303ed5f6ac43b7866d8efd0313bdc2: Time spent bootstrapping tablet: real 0.011s user 0.006s sys 0.003s
I20260620 14:48:09.711187 7913 raft_consensus.cc:348] T 77de1104f698409ab9e3664c1465711f P 76303ed5f6ac43b7866d8efd0313bdc2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "76303ed5f6ac43b7866d8efd0313bdc2" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 39249 } } peers { permanent_uuid: "790f4a4136784937b4bfc2572c01a54b" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 36327 } } peers { permanent_uuid: "361b2f75b32a48cda1d69fd295208d8d" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 46083 } }
I20260620 14:48:09.711474 7913 raft_consensus.cc:374] T 77de1104f698409ab9e3664c1465711f P 76303ed5f6ac43b7866d8efd0313bdc2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 14:48:09.711649 7913 raft_consensus.cc:729] T 77de1104f698409ab9e3664c1465711f P 76303ed5f6ac43b7866d8efd0313bdc2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 76303ed5f6ac43b7866d8efd0313bdc2, State: Initialized, Role: FOLLOWER
I20260620 14:48:09.713124 7910 raft_consensus.cc:482] T 77de1104f698409ab9e3664c1465711f P 790f4a4136784937b4bfc2572c01a54b [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260620 14:48:09.713111 7913 consensus_queue.cc:260] T 77de1104f698409ab9e3664c1465711f P 76303ed5f6ac43b7866d8efd0313bdc2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "76303ed5f6ac43b7866d8efd0313bdc2" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 39249 } } peers { permanent_uuid: "790f4a4136784937b4bfc2572c01a54b" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 36327 } } peers { permanent_uuid: "361b2f75b32a48cda1d69fd295208d8d" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 46083 } }
I20260620 14:48:09.713315 7910 raft_consensus.cc:504] T 77de1104f698409ab9e3664c1465711f P 790f4a4136784937b4bfc2572c01a54b [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "76303ed5f6ac43b7866d8efd0313bdc2" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 39249 } } peers { permanent_uuid: "790f4a4136784937b4bfc2572c01a54b" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 36327 } } peers { permanent_uuid: "361b2f75b32a48cda1d69fd295208d8d" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 46083 } }
I20260620 14:48:09.713724 7912 ts_tablet_manager.cc:1428] T 77de1104f698409ab9e3664c1465711f P 361b2f75b32a48cda1d69fd295208d8d: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20260620 14:48:09.713964 7815 heartbeater.cc:503] Master 127.0.97.61:41719 was elected leader, sending a full tablet report...
I20260620 14:48:09.715278 7910 leader_election.cc:288] T 77de1104f698409ab9e3664c1465711f P 790f4a4136784937b4bfc2572c01a54b [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 76303ed5f6ac43b7866d8efd0313bdc2 (127.0.97.1:39249), 361b2f75b32a48cda1d69fd295208d8d (127.0.97.3:46083)
I20260620 14:48:09.715473 7917 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:48:09.711278 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:39515 (local address 127.0.97.3:40069)
0620 14:48:09.715068 (+ 3790us) negotiation.cc:107] Waiting for socket to connect
0620 14:48:09.715205 (+ 137us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:39515: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":3662,"thread_start_us":3505,"threads_started":1}
I20260620 14:48:09.720588 7920 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:48:09.716025 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:39515 (local address 127.0.97.3:42967)
0620 14:48:09.720072 (+ 4047us) negotiation.cc:107] Waiting for socket to connect
0620 14:48:09.720268 (+ 196us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:39515: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":3902,"thread_start_us":3791,"threads_started":1}
I20260620 14:48:09.722587 7913 ts_tablet_manager.cc:1428] T 77de1104f698409ab9e3664c1465711f P 76303ed5f6ac43b7866d8efd0313bdc2: Time spent starting tablet: real 0.012s user 0.005s sys 0.000s
I20260620 14:48:09.724557 7527 heartbeater.cc:503] Master 127.0.97.61:41719 was elected leader, sending a full tablet report...
I20260620 14:48:09.727664 7904 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:48:09.716004 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.2:39635 (local address 127.0.97.1:39249)
0620 14:48:09.716203 (+ 199us) server_negotiation.cc:206] Beginning negotiation
0620 14:48:09.716210 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0620 14:48:09.716503 (+ 293us) server_negotiation.cc:407] Connection header received
0620 14:48:09.716711 (+ 208us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:48:09.716716 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:48:09.716797 (+ 81us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:48:09.716886 (+ 89us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 14:48:09.717591 (+ 705us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:48:09.718897 (+ 1306us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:48:09.725524 (+ 6627us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:48:09.726639 (+ 1115us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:48:09.726797 (+ 158us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:48:09.726931 (+ 134us) server_negotiation.cc:1035] Waiting for connection context
0620 14:48:09.727332 (+ 401us) server_negotiation.cc:299] Negotiation successful
0620 14:48:09.727423 (+ 91us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":57}
I20260620 14:48:09.728417 7911 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:48:09.715489 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:39249 (local address 127.0.97.2:39635)
0620 14:48:09.716401 (+ 912us) negotiation.cc:107] Waiting for socket to connect
0620 14:48:09.716416 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0620 14:48:09.716581 (+ 165us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 14:48:09.716941 (+ 360us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 14:48:09.716949 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 14:48:09.716989 (+ 40us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 14:48:09.717366 (+ 377us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:48:09.717380 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:48:09.719066 (+ 1686us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:48:09.719075 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:48:09.725340 (+ 6265us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:48:09.725365 (+ 25us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:48:09.726839 (+ 1474us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:48:09.726848 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:48:09.727059 (+ 211us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:48:09.727100 (+ 41us) client_negotiation.cc:769] Sending connection context
0620 14:48:09.727195 (+ 95us) client_negotiation.cc:240] Negotiation successful
0620 14:48:09.727371 (+ 176us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":817}
I20260620 14:48:09.729300 7480 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "77de1104f698409ab9e3664c1465711f" candidate_uuid: "790f4a4136784937b4bfc2572c01a54b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "76303ed5f6ac43b7866d8efd0313bdc2" is_pre_election: true
I20260620 14:48:09.729846 7480 raft_consensus.cc:2454] T 77de1104f698409ab9e3664c1465711f P 76303ed5f6ac43b7866d8efd0313bdc2 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 790f4a4136784937b4bfc2572c01a54b in term 0.
I20260620 14:48:09.730643 7560 leader_election.cc:302] T 77de1104f698409ab9e3664c1465711f P 790f4a4136784937b4bfc2572c01a54b [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 76303ed5f6ac43b7866d8efd0313bdc2, 790f4a4136784937b4bfc2572c01a54b; no voters:
I20260620 14:48:09.731209 7910 raft_consensus.cc:2781] T 77de1104f698409ab9e3664c1465711f P 790f4a4136784937b4bfc2572c01a54b [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260620 14:48:09.731354 7910 raft_consensus.cc:482] T 77de1104f698409ab9e3664c1465711f P 790f4a4136784937b4bfc2572c01a54b [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260620 14:48:09.731463 7910 raft_consensus.cc:3037] T 77de1104f698409ab9e3664c1465711f P 790f4a4136784937b4bfc2572c01a54b [term 0 FOLLOWER]: Advancing to term 1
I20260620 14:48:09.731786 7906 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:48:09.716244 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.2:35785 (local address 127.0.97.3:46083)
0620 14:48:09.716446 (+ 202us) server_negotiation.cc:206] Beginning negotiation
0620 14:48:09.716456 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0620 14:48:09.726753 (+ 10297us) server_negotiation.cc:407] Connection header received
0620 14:48:09.726930 (+ 177us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:48:09.726935 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:48:09.727025 (+ 90us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:48:09.727112 (+ 87us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 14:48:09.727675 (+ 563us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:48:09.728439 (+ 764us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:48:09.730053 (+ 1614us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:48:09.730820 (+ 767us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:48:09.730975 (+ 155us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:48:09.731086 (+ 111us) server_negotiation.cc:1035] Waiting for connection context
0620 14:48:09.731387 (+ 301us) server_negotiation.cc:299] Negotiation successful
0620 14:48:09.731510 (+ 123us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":38}
I20260620 14:48:09.731874 7919 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:48:09.715719 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.3:46083 (local address 127.0.97.2:35785)
0620 14:48:09.726609 (+ 10890us) negotiation.cc:107] Waiting for socket to connect
0620 14:48:09.726634 (+ 25us) client_negotiation.cc:174] Beginning negotiation
0620 14:48:09.726790 (+ 156us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 14:48:09.727119 (+ 329us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 14:48:09.727124 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 14:48:09.727154 (+ 30us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 14:48:09.727488 (+ 334us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:48:09.727499 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:48:09.728575 (+ 1076us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:48:09.728581 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:48:09.729902 (+ 1321us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:48:09.729915 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:48:09.731049 (+ 1134us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:48:09.731056 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:48:09.731227 (+ 171us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:48:09.731250 (+ 23us) client_negotiation.cc:769] Sending connection context
0620 14:48:09.731362 (+ 112us) client_negotiation.cc:240] Negotiation successful
0620 14:48:09.731495 (+ 133us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":10720,"thread_start_us":11169,"threads_started":1}
I20260620 14:48:09.735679 7910 raft_consensus.cc:504] T 77de1104f698409ab9e3664c1465711f P 790f4a4136784937b4bfc2572c01a54b [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "76303ed5f6ac43b7866d8efd0313bdc2" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 39249 } } peers { permanent_uuid: "790f4a4136784937b4bfc2572c01a54b" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 36327 } } peers { permanent_uuid: "361b2f75b32a48cda1d69fd295208d8d" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 46083 } }
I20260620 14:48:09.736398 7921 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:48:09.721906 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:39515 (local address 127.0.97.1:42883)
0620 14:48:09.735711 (+ 13805us) negotiation.cc:107] Waiting for socket to connect
0620 14:48:09.736084 (+ 373us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:39515: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":13657,"thread_start_us":13517,"threads_started":1}
I20260620 14:48:09.736353 7768 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "77de1104f698409ab9e3664c1465711f" candidate_uuid: "790f4a4136784937b4bfc2572c01a54b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "361b2f75b32a48cda1d69fd295208d8d" is_pre_election: true
I20260620 14:48:09.736608 7768 raft_consensus.cc:2454] T 77de1104f698409ab9e3664c1465711f P 361b2f75b32a48cda1d69fd295208d8d [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 790f4a4136784937b4bfc2572c01a54b in term 0.
I20260620 14:48:09.736824 7910 leader_election.cc:288] T 77de1104f698409ab9e3664c1465711f P 790f4a4136784937b4bfc2572c01a54b [CANDIDATE]: Term 1 election: Requested vote from peers 76303ed5f6ac43b7866d8efd0313bdc2 (127.0.97.1:39249), 361b2f75b32a48cda1d69fd295208d8d (127.0.97.3:46083)
I20260620 14:48:09.736771 7480 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "77de1104f698409ab9e3664c1465711f" candidate_uuid: "790f4a4136784937b4bfc2572c01a54b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "76303ed5f6ac43b7866d8efd0313bdc2"
I20260620 14:48:09.736976 7480 raft_consensus.cc:3037] T 77de1104f698409ab9e3664c1465711f P 76303ed5f6ac43b7866d8efd0313bdc2 [term 0 FOLLOWER]: Advancing to term 1
I20260620 14:48:09.737407 7768 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "77de1104f698409ab9e3664c1465711f" candidate_uuid: "790f4a4136784937b4bfc2572c01a54b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "361b2f75b32a48cda1d69fd295208d8d"
I20260620 14:48:09.737574 7768 raft_consensus.cc:3037] T 77de1104f698409ab9e3664c1465711f P 361b2f75b32a48cda1d69fd295208d8d [term 0 FOLLOWER]: Advancing to term 1
I20260620 14:48:09.737751 7921 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:48:09.737276 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:39515 (local address 127.0.97.1:40305)
0620 14:48:09.737496 (+ 220us) negotiation.cc:107] Waiting for socket to connect
0620 14:48:09.737587 (+ 91us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:39515: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":125}
I20260620 14:48:09.739569 7480 raft_consensus.cc:2454] T 77de1104f698409ab9e3664c1465711f P 76303ed5f6ac43b7866d8efd0313bdc2 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 790f4a4136784937b4bfc2572c01a54b in term 1.
I20260620 14:48:09.739985 7560 leader_election.cc:302] T 77de1104f698409ab9e3664c1465711f P 790f4a4136784937b4bfc2572c01a54b [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 76303ed5f6ac43b7866d8efd0313bdc2, 790f4a4136784937b4bfc2572c01a54b; no voters:
I20260620 14:48:09.740254 7910 raft_consensus.cc:2781] T 77de1104f698409ab9e3664c1465711f P 790f4a4136784937b4bfc2572c01a54b [term 1 FOLLOWER]: Leader election won for term 1
I20260620 14:48:09.740263 7768 raft_consensus.cc:2454] T 77de1104f698409ab9e3664c1465711f P 361b2f75b32a48cda1d69fd295208d8d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 790f4a4136784937b4bfc2572c01a54b in term 1.
I20260620 14:48:09.741060 7910 raft_consensus.cc:686] T 77de1104f698409ab9e3664c1465711f P 790f4a4136784937b4bfc2572c01a54b [term 1 LEADER]: Becoming Leader. State: Replica: 790f4a4136784937b4bfc2572c01a54b, State: Running, Role: LEADER
I20260620 14:48:09.741492 7910 consensus_queue.cc:237] T 77de1104f698409ab9e3664c1465711f P 790f4a4136784937b4bfc2572c01a54b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "76303ed5f6ac43b7866d8efd0313bdc2" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 39249 } } peers { permanent_uuid: "790f4a4136784937b4bfc2572c01a54b" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 36327 } } peers { permanent_uuid: "361b2f75b32a48cda1d69fd295208d8d" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 46083 } }
I20260620 14:48:09.741922 7919 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:48:09.741568 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:39515 (local address 127.0.97.2:44897)
0620 14:48:09.741713 (+ 145us) negotiation.cc:107] Waiting for socket to connect
0620 14:48:09.741782 (+ 69us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:39515: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":33}
I20260620 14:48:09.744935 7149 catalog_manager.cc:5495] T 77de1104f698409ab9e3664c1465711f P 790f4a4136784937b4bfc2572c01a54b reported cstate change: term changed from 0 to 1, leader changed from <none> to 790f4a4136784937b4bfc2572c01a54b (127.0.97.2). New cstate: current_term: 1 leader_uuid: "790f4a4136784937b4bfc2572c01a54b" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "76303ed5f6ac43b7866d8efd0313bdc2" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 39249 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "790f4a4136784937b4bfc2572c01a54b" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 36327 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "361b2f75b32a48cda1d69fd295208d8d" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 46083 } health_report { overall_health: UNKNOWN } } }
I20260620 14:48:09.763027 7907 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:48:09.758027 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40076 (local address 127.0.97.2:36327)
0620 14:48:09.758214 (+ 187us) server_negotiation.cc:206] Beginning negotiation
0620 14:48:09.758219 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 14:48:09.758236 (+ 17us) server_negotiation.cc:407] Connection header received
0620 14:48:09.758519 (+ 283us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:48:09.758524 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:48:09.758637 (+ 113us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:48:09.758774 (+ 137us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 14:48:09.759072 (+ 298us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:48:09.759871 (+ 799us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:48:09.761078 (+ 1207us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:48:09.761494 (+ 416us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:48:09.761596 (+ 102us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:48:09.761934 (+ 338us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 14:48:09.762462 (+ 528us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0620 14:48:09.762605 (+ 143us) server_negotiation.cc:1035] Waiting for connection context
0620 14:48:09.762649 (+ 44us) server_negotiation.cc:299] Negotiation successful
0620 14:48:09.762713 (+ 64us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":65}
I20260620 14:48:09.766945 7768 raft_consensus.cc:1261] T 77de1104f698409ab9e3664c1465711f P 361b2f75b32a48cda1d69fd295208d8d [term 1 FOLLOWER]: Refusing update from remote peer 790f4a4136784937b4bfc2572c01a54b: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 14:48:09.767438 7480 raft_consensus.cc:1261] T 77de1104f698409ab9e3664c1465711f P 76303ed5f6ac43b7866d8efd0313bdc2 [term 1 FOLLOWER]: Refusing update from remote peer 790f4a4136784937b4bfc2572c01a54b: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 14:48:09.767750 7922 consensus_queue.cc:1035] T 77de1104f698409ab9e3664c1465711f P 790f4a4136784937b4bfc2572c01a54b [LEADER]: Connected to new peer: Peer: permanent_uuid: "361b2f75b32a48cda1d69fd295208d8d" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 46083 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 14:48:09.768249 7910 consensus_queue.cc:1035] T 77de1104f698409ab9e3664c1465711f P 790f4a4136784937b4bfc2572c01a54b [LEADER]: Connected to new peer: Peer: permanent_uuid: "76303ed5f6ac43b7866d8efd0313bdc2" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 39249 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 14:48:09.770493 7917 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:48:09.769767 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:39515 (local address 127.0.97.3:53275)
0620 14:48:09.770251 (+ 484us) negotiation.cc:107] Waiting for socket to connect
0620 14:48:09.770345 (+ 94us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:39515: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":382}
I20260620 14:48:09.774793 7925 mvcc.cc:205] Tried to move back new op lower bound from 7298936380479049728 to 7298936380385153024. Current Snapshot: MvccSnapshot[applied={T|T < 7298936380479049728}]
I20260620 14:48:09.775139 7926 mvcc.cc:205] Tried to move back new op lower bound from 7298936380479049728 to 7298936380385153024. Current Snapshot: MvccSnapshot[applied={T|T < 7298936380479049728}]
I20260620 14:48:09.777006 7917 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:48:09.776291 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:39515 (local address 127.0.97.3:38547)
0620 14:48:09.776617 (+ 326us) negotiation.cc:107] Waiting for socket to connect
0620 14:48:09.776749 (+ 132us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:39515: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":188}
I20260620 14:48:09.778056 7921 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:48:09.771144 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:39515 (local address 127.0.97.1:49313)
0620 14:48:09.771459 (+ 315us) negotiation.cc:107] Waiting for socket to connect
0620 14:48:09.771611 (+ 152us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:39515: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":169}
I20260620 14:48:09.781944 7919 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:48:09.781498 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:39515 (local address 127.0.97.2:38269)
0620 14:48:09.781684 (+ 186us) negotiation.cc:107] Waiting for socket to connect
0620 14:48:09.781785 (+ 101us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:39515: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":74}
I20260620 14:48:09.789598 7919 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:48:09.789084 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:39515 (local address 127.0.97.2:50081)
0620 14:48:09.789314 (+ 230us) negotiation.cc:107] Waiting for socket to connect
0620 14:48:09.789405 (+ 91us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:39515: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":125}
I20260620 14:48:09.789865 7921 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:48:09.786722 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:39515 (local address 127.0.97.1:34963)
0620 14:48:09.789424 (+ 2702us) negotiation.cc:107] Waiting for socket to connect
0620 14:48:09.789575 (+ 151us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:39515: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":2552}
I20260620 14:48:09.801064 7149 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:48096:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 14:48:09.801410 7149 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:48096:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 14:48:09.806829 7149 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 55c0ce5da3694b759b5b3113647c3eaa: Sending DeleteTablet for 3 replicas of tablet 77de1104f698409ab9e3664c1465711f
I20260620 14:48:09.807977 7608 tablet_service.cc:1514] Processing DeleteTablet for tablet 77de1104f698409ab9e3664c1465711f with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 14:48:09 UTC) from {username='oryx', principal='oryx/127.0.97.61@KRBTEST.COM'} at 127.0.0.1:40074
I20260620 14:48:09.808063 7460 tablet_service.cc:1514] Processing DeleteTablet for tablet 77de1104f698409ab9e3664c1465711f with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 14:48:09 UTC) from {username='oryx', principal='oryx/127.0.97.61@KRBTEST.COM'} at 127.0.0.1:52102
I20260620 14:48:09.808389 7748 tablet_service.cc:1514] Processing DeleteTablet for tablet 77de1104f698409ab9e3664c1465711f with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 14:48:09 UTC) from {username='oryx', principal='oryx/127.0.97.61@KRBTEST.COM'} at 127.0.0.1:41754
I20260620 14:48:09.810314 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskUci3DU/build/asan/bin/kudu pid 7312
I20260620 14:48:09.822739 7937 tablet_replica.cc:330] T 77de1104f698409ab9e3664c1465711f P 361b2f75b32a48cda1d69fd295208d8d: stopping tablet replica
I20260620 14:48:09.823000 7937 raft_consensus.cc:2229] T 77de1104f698409ab9e3664c1465711f P 361b2f75b32a48cda1d69fd295208d8d [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 14:48:09.823112 7936 tablet_replica.cc:330] T 77de1104f698409ab9e3664c1465711f P 76303ed5f6ac43b7866d8efd0313bdc2: stopping tablet replica
I20260620 14:48:09.823366 7937 raft_consensus.cc:2258] T 77de1104f698409ab9e3664c1465711f P 361b2f75b32a48cda1d69fd295208d8d [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 14:48:09.823401 7936 raft_consensus.cc:2229] T 77de1104f698409ab9e3664c1465711f P 76303ed5f6ac43b7866d8efd0313bdc2 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 14:48:09.823755 7936 raft_consensus.cc:2258] T 77de1104f698409ab9e3664c1465711f P 76303ed5f6ac43b7866d8efd0313bdc2 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 14:48:09.827476 7935 tablet_replica.cc:330] T 77de1104f698409ab9e3664c1465711f P 790f4a4136784937b4bfc2572c01a54b: stopping tablet replica
I20260620 14:48:09.827711 7935 raft_consensus.cc:2229] T 77de1104f698409ab9e3664c1465711f P 790f4a4136784937b4bfc2572c01a54b [term 1 LEADER]: Raft consensus shutting down.
I20260620 14:48:09.828346 7935 raft_consensus.cc:2258] T 77de1104f698409ab9e3664c1465711f P 790f4a4136784937b4bfc2572c01a54b [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 14:48:09.828677 7936 ts_tablet_manager.cc:1905] T 77de1104f698409ab9e3664c1465711f P 76303ed5f6ac43b7866d8efd0313bdc2: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 14:48:09.829373 7935 ts_tablet_manager.cc:1905] T 77de1104f698409ab9e3664c1465711f P 790f4a4136784937b4bfc2572c01a54b: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 14:48:09.829375 7937 ts_tablet_manager.cc:1905] T 77de1104f698409ab9e3664c1465711f P 361b2f75b32a48cda1d69fd295208d8d: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 14:48:09.836153 7935 ts_tablet_manager.cc:1918] T 77de1104f698409ab9e3664c1465711f P 790f4a4136784937b4bfc2572c01a54b: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260620 14:48:09.836153 7937 ts_tablet_manager.cc:1918] T 77de1104f698409ab9e3664c1465711f P 361b2f75b32a48cda1d69fd295208d8d: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260620 14:48:09.836303 7935 log.cc:1196] T 77de1104f698409ab9e3664c1465711f P 790f4a4136784937b4bfc2572c01a54b: Deleting WAL directory at /tmp/dist-test-taskUci3DU/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781966829091300-388-0/minicluster-data/ts-1/wal/wals/77de1104f698409ab9e3664c1465711f
I20260620 14:48:09.836301 7937 log.cc:1196] T 77de1104f698409ab9e3664c1465711f P 361b2f75b32a48cda1d69fd295208d8d: Deleting WAL directory at /tmp/dist-test-taskUci3DU/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781966829091300-388-0/minicluster-data/ts-2/wal/wals/77de1104f698409ab9e3664c1465711f
I20260620 14:48:09.836743 7937 ts_tablet_manager.cc:1939] T 77de1104f698409ab9e3664c1465711f P 361b2f75b32a48cda1d69fd295208d8d: Deleting consensus metadata
I20260620 14:48:09.836747 7935 ts_tablet_manager.cc:1939] T 77de1104f698409ab9e3664c1465711f P 790f4a4136784937b4bfc2572c01a54b: Deleting consensus metadata
I20260620 14:48:09.837515 7936 ts_tablet_manager.cc:1918] T 77de1104f698409ab9e3664c1465711f P 76303ed5f6ac43b7866d8efd0313bdc2: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260620 14:48:09.837975 7134 catalog_manager.cc:4841] TS 361b2f75b32a48cda1d69fd295208d8d (127.0.97.3:46083): tablet 77de1104f698409ab9e3664c1465711f (table test-table [id=27bc53d06b394ae5a47a19167b6830e1]) successfully deleted
I20260620 14:48:09.838371 7135 catalog_manager.cc:4841] TS 790f4a4136784937b4bfc2572c01a54b (127.0.97.2:36327): tablet 77de1104f698409ab9e3664c1465711f (table test-table [id=27bc53d06b394ae5a47a19167b6830e1]) successfully deleted
I20260620 14:48:09.838575 7936 log.cc:1196] T 77de1104f698409ab9e3664c1465711f P 76303ed5f6ac43b7866d8efd0313bdc2: Deleting WAL directory at /tmp/dist-test-taskUci3DU/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781966829091300-388-0/minicluster-data/ts-0/wal/wals/77de1104f698409ab9e3664c1465711f
I20260620 14:48:09.839143 7936 ts_tablet_manager.cc:1939] T 77de1104f698409ab9e3664c1465711f P 76303ed5f6ac43b7866d8efd0313bdc2: Deleting consensus metadata
I20260620 14:48:09.840517 7133 catalog_manager.cc:4841] TS 76303ed5f6ac43b7866d8efd0313bdc2 (127.0.97.1:39249): tablet 77de1104f698409ab9e3664c1465711f (table test-table [id=27bc53d06b394ae5a47a19167b6830e1]) successfully deleted
I20260620 14:48:09.841890 7904 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:48:09.812752 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52118 (local address 127.0.97.1:39249)
0620 14:48:09.813016 (+ 264us) server_negotiation.cc:206] Beginning negotiation
0620 14:48:09.813027 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0620 14:48:09.824489 (+ 11462us) server_negotiation.cc:407] Connection header received
0620 14:48:09.824688 (+ 199us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:48:09.824697 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:48:09.824825 (+ 128us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:48:09.824986 (+ 161us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:48:09.827058 (+ 2072us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:48:09.828011 (+ 953us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:48:09.830009 (+ 1998us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:48:09.830428 (+ 419us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:48:09.830583 (+ 155us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:48:09.833978 (+ 3395us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 14:48:09.833994 (+ 16us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 14:48:09.834011 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 14:48:09.834047 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0620 14:48:09.838101 (+ 4054us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:48:09.839146 (+ 1045us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:48:09.839158 (+ 12us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:48:09.839165 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:48:09.839384 (+ 219us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:48:09.839932 (+ 548us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:48:09.839939 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:48:09.839944 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:48:09.840438 (+ 494us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 14:48:09.840685 (+ 247us) server_negotiation.cc:1035] Waiting for connection context
0620 14:48:09.841326 (+ 641us) server_negotiation.cc:299] Negotiation successful
0620 14:48:09.841547 (+ 221us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":65}
I20260620 14:48:09.842653 7520 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 14:48:09.960896 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskUci3DU/build/asan/bin/kudu with pid 7312
I20260620 14:48:09.984210 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskUci3DU/build/asan/bin/kudu pid 7538
I20260620 14:48:09.998534 7907 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:48:09.985206 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40080 (local address 127.0.97.2:36327)
0620 14:48:09.985405 (+ 199us) server_negotiation.cc:206] Beginning negotiation
0620 14:48:09.985415 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0620 14:48:09.985439 (+ 24us) server_negotiation.cc:407] Connection header received
0620 14:48:09.985492 (+ 53us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:48:09.985495 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:48:09.985550 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:48:09.985694 (+ 144us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:48:09.987066 (+ 1372us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:48:09.987957 (+ 891us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:48:09.989189 (+ 1232us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:48:09.989630 (+ 441us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:48:09.989793 (+ 163us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:48:09.992383 (+ 2590us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 14:48:09.992398 (+ 15us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 14:48:09.992415 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 14:48:09.992450 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0620 14:48:09.995722 (+ 3272us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:48:09.996398 (+ 676us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:48:09.996410 (+ 12us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:48:09.996418 (+ 8us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:48:09.996594 (+ 176us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:48:09.996981 (+ 387us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:48:09.996987 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:48:09.996991 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:48:09.997332 (+ 341us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 14:48:09.997638 (+ 306us) server_negotiation.cc:1035] Waiting for connection context
0620 14:48:09.997947 (+ 309us) server_negotiation.cc:299] Negotiation successful
0620 14:48:09.998167 (+ 220us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":69}
I20260620 14:48:09.999179 7668 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 14:48:10.120900 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskUci3DU/build/asan/bin/kudu with pid 7538
I20260620 14:48:10.131718 7905 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:48:10.131200 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:39515 (local address 127.0.0.1:43214)
0620 14:48:10.131387 (+ 187us) negotiation.cc:107] Waiting for socket to connect
0620 14:48:10.131505 (+ 118us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:39515: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":46}
I20260620 14:48:10.136853 7905 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:48:10.136415 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:39515 (local address 127.0.0.1:43222)
0620 14:48:10.136579 (+ 164us) negotiation.cc:107] Waiting for socket to connect
0620 14:48:10.136697 (+ 118us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:39515: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":53}
I20260620 14:48:10.148129 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskUci3DU/build/asan/bin/kudu pid 7685
I20260620 14:48:10.163010 7906 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:48:10.148874 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41770 (local address 127.0.97.3:46083)
0620 14:48:10.149056 (+ 182us) server_negotiation.cc:206] Beginning negotiation
0620 14:48:10.149063 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0620 14:48:10.149115 (+ 52us) server_negotiation.cc:407] Connection header received
0620 14:48:10.149300 (+ 185us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:48:10.149305 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:48:10.149391 (+ 86us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:48:10.149494 (+ 103us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:48:10.150765 (+ 1271us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:48:10.151967 (+ 1202us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:48:10.153146 (+ 1179us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:48:10.153810 (+ 664us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:48:10.153942 (+ 132us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:48:10.156604 (+ 2662us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 14:48:10.156629 (+ 25us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 14:48:10.156650 (+ 21us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 14:48:10.156693 (+ 43us) server_negotiation.cc:922] calling sasl_server_start()
0620 14:48:10.160260 (+ 3567us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:48:10.160993 (+ 733us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:48:10.161002 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:48:10.161006 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:48:10.161163 (+ 157us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:48:10.161519 (+ 356us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:48:10.161524 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:48:10.161528 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:48:10.161966 (+ 438us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 14:48:10.162245 (+ 279us) server_negotiation.cc:1035] Waiting for connection context
0620 14:48:10.162425 (+ 180us) server_negotiation.cc:299] Negotiation successful
0620 14:48:10.162662 (+ 237us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":67}
I20260620 14:48:10.163774 7808 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 14:48:10.257530 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskUci3DU/build/asan/bin/kudu with pid 7685
I20260620 14:48:10.274606 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskUci3DU/build/asan/bin/kudu pid 7012
I20260620 14:48:10.275841 7179 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 14:48:10.397105 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskUci3DU/build/asan/bin/kudu with pid 7012
I20260620 14:48:10.418867 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskUci3DU/build/asan/bin/kudu pid 7112
I20260620 14:48:10.419994 7378 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 14:48:10.534852 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskUci3DU/build/asan/bin/kudu with pid 7112
2026-06-20T14:48:10Z chronyd exiting
[ OK ] SecurityITest.TestNonDefaultPrincipalMultipleMaster (9761 ms)
[ RUN ] SecurityITest.TestNonDefaultPrincipal
Loading random data
Initializing database '/tmp/dist-test-taskUci3DU/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781966829091300-388-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 20 14:48:10 dist-test-slave-q9j6 krb5kdc[7946](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 20 14:48:10 dist-test-slave-q9j6 krb5kdc[7946](info): set up 2 sockets
Jun 20 14:48:10 dist-test-slave-q9j6 krb5kdc[7946](info): commencing operation
krb5kdc: starting...
W20260620 14:48:11.632503 388 mini_kdc.cc:121] Time spent starting KDC: real 1.052s user 0.002s sys 0.019s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskUci3DU/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781966829091300-388-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskUci3DU/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781966829091300-388-0/krb5kdc/test-user.keytab.
Jun 20 14:48:11 dist-test-slave-q9j6 krb5kdc[7946](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781966891, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-20T14:48:11Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T14:48:11Z Disabled control of system clock
WARNING: no policy specified for oryx/127.0.97.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.0.97.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.0.97.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskUci3DU/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781966829091300-388-0/krb5kdc/oryx_127.0.97.62.keytab.
Entry for principal oryx/127.0.97.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskUci3DU/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781966829091300-388-0/krb5kdc/oryx_127.0.97.62.keytab.
I20260620 14:48:11.853289 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskUci3DU/build/asan/bin/kudu
/tmp/dist-test-taskUci3DU/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskUci3DU/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781966829091300-388-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskUci3DU/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781966829091300-388-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskUci3DU/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781966829091300-388-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskUci3DU/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781966829091300-388-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:33203
--webserver_interface=127.0.97.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskUci3DU/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781966829091300-388-0/krb5kdc/oryx_127.0.97.62.keytab
--principal=oryx/127.0.97.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:43131
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.97.62:33203
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--txn_manager_enabled=true with env {KRB5CCNAME=/tmp/dist-test-taskUci3DU/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781966829091300-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskUci3DU/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781966829091300-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskUci3DU/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781966829091300-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 14:48:12.186961 7959 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 14:48:12.187469 7959 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 14:48:12.187637 7959 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 14:48:12.195654 7959 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 14:48:12.195819 7959 flags.cc:425] Enabled experimental flag: --txn_manager_enabled=true
W20260620 14:48:12.195894 7959 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 14:48:12.195989 7959 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 14:48:12.196084 7959 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 14:48:12.196172 7959 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 14:48:12.206326 7959 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:43131
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskUci3DU/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781966829091300-388-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskUci3DU/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781966829091300-388-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.97.62:33203
--txn_manager_enabled=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskUci3DU/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781966829091300-388-0/krb5kdc/oryx_127.0.97.62.keytab
--principal=oryx/127.0.97.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:33203
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskUci3DU/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781966829091300-388-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.97.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskUci3DU/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781966829091300-388-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 13:41:20 UTC on ccycloud.quasar-vdagoi.root.comops.site
build id 4457462
ASAN enabled
I20260620 14:48:12.207770 7959 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 14:48:12.209201 7959 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 14:48:12.221521 7964 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 14:48:12.221529 7965 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 14:48:12.221793 7967 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 14:48:12.222270 7959 server_base.cc:1034] running on GCE node
I20260620 14:48:12.223165 7959 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 14:48:12.224750 7959 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 14:48:12.226001 7959 hybrid_clock.cc:648] HybridClock initialized: now 1781966892225920 us; error 63 us; skew 500 ppm
I20260620 14:48:12.226354 7959 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 14:48:12 dist-test-slave-q9j6 krb5kdc[7946](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781966892, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.97.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 14:48:12.229599 7959 init.cc:377] Logged in from keytab as oryx/127.0.97.62@KRBTEST.COM (short username oryx)
I20260620 14:48:12.231158 7959 webserver.cc:460] Webserver started at http://127.0.97.62:43167/ using document root <none> and password file <none>
I20260620 14:48:12.232005 7959 fs_manager.cc:359] Metadata directory not provided
I20260620 14:48:12.232141 7959 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 14:48:12.232484 7959 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 14:48:12.235657 7959 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskUci3DU/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781966829091300-388-0/minicluster-data/master-0/data/instance:
uuid: "0af6e58150ce4b73a4c412dd6c640a60"
format_stamp: "Formatted at 2026-06-20 14:48:12 on dist-test-slave-q9j6"
server_key: "ed2e0842aab409b1be736b70927f4dac"
server_key_iv: "bfe3e08222bb20198205682241dbe621"
server_key_version: "encryptionkey@0"
I20260620 14:48:12.236547 7959 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskUci3DU/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781966829091300-388-0/minicluster-data/master-0/wal/instance:
uuid: "0af6e58150ce4b73a4c412dd6c640a60"
format_stamp: "Formatted at 2026-06-20 14:48:12 on dist-test-slave-q9j6"
server_key: "ed2e0842aab409b1be736b70927f4dac"
server_key_iv: "bfe3e08222bb20198205682241dbe621"
server_key_version: "encryptionkey@0"
I20260620 14:48:12.242074 7959 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.001s sys 0.004s
I20260620 14:48:12.245894 7974 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 14:48:12.247340 7959 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260620 14:48:12.247542 7959 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskUci3DU/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781966829091300-388-0/minicluster-data/master-0/data,/tmp/dist-test-taskUci3DU/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781966829091300-388-0/minicluster-data/master-0/wal
uuid: "0af6e58150ce4b73a4c412dd6c640a60"
format_stamp: "Formatted at 2026-06-20 14:48:12 on dist-test-slave-q9j6"
server_key: "ed2e0842aab409b1be736b70927f4dac"
server_key_iv: "bfe3e08222bb20198205682241dbe621"
server_key_version: "encryptionkey@0"
I20260620 14:48:12.247826 7959 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskUci3DU/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781966829091300-388-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskUci3DU/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781966829091300-388-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskUci3DU/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781966829091300-388-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 14:48:12.265309 7959 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 14:48:12.269529 7959 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 14:48:12.269824 7959 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 14:48:12.292268 7959 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.62:33203
I20260620 14:48:12.292308 8035 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.62:33203 every 8 connection(s)
I20260620 14:48:12.293777 7959 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskUci3DU/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781966829091300-388-0/minicluster-data/master-0/data/info.pb
I20260620 14:48:12.294955 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskUci3DU/build/asan/bin/kudu as pid 7959
I20260620 14:48:12.295200 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskUci3DU/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781966829091300-388-0/minicluster-data/master-0/wal/instance
I20260620 14:48:12.296015 388 external_mini_cluster.cc:1428] Setting key c7042268809e239b9459415ab8556786
I20260620 14:48:12.298966 8036 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
Jun 20 14:48:12 dist-test-slave-q9j6 krb5kdc[7946](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781966891, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM
I20260620 14:48:12.310774 8036 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 0af6e58150ce4b73a4c412dd6c640a60: Bootstrap starting.
I20260620 14:48:12.314375 8036 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 0af6e58150ce4b73a4c412dd6c640a60: Neither blocks nor log segments found. Creating new log.
I20260620 14:48:12.315469 8036 log.cc:824] T 00000000000000000000000000000000 P 0af6e58150ce4b73a4c412dd6c640a60: Log is configured to *not* fsync() on all Append() calls
I20260620 14:48:12.318502 8036 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 0af6e58150ce4b73a4c412dd6c640a60: No bootstrap required, opened a new log
I20260620 14:48:12.319099 8039 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:48:12.298164 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52186 (local address 127.0.97.62:33203)
0620 14:48:12.298995 (+ 831us) server_negotiation.cc:206] Beginning negotiation
0620 14:48:12.299008 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0620 14:48:12.299088 (+ 80us) server_negotiation.cc:407] Connection header received
0620 14:48:12.300826 (+ 1738us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:48:12.300850 (+ 24us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:48:12.301327 (+ 477us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:48:12.301815 (+ 488us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:48:12.303999 (+ 2184us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:48:12.305397 (+ 1398us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:48:12.306667 (+ 1270us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:48:12.307364 (+ 697us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:48:12.307626 (+ 262us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:48:12.311762 (+ 4136us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 14:48:12.311771 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 14:48:12.311786 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 14:48:12.311823 (+ 37us) server_negotiation.cc:922] calling sasl_server_start()
0620 14:48:12.314919 (+ 3096us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:48:12.315726 (+ 807us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:48:12.315735 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:48:12.315742 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:48:12.315922 (+ 180us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:48:12.316349 (+ 427us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:48:12.316355 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:48:12.316363 (+ 8us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:48:12.316890 (+ 527us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 14:48:12.317177 (+ 287us) server_negotiation.cc:1035] Waiting for connection context
0620 14:48:12.317629 (+ 452us) server_negotiation.cc:299] Negotiation successful
0620 14:48:12.318038 (+ 409us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":615,"thread_start_us":399,"threads_started":1}
I20260620 14:48:12.326732 8036 raft_consensus.cc:348] T 00000000000000000000000000000000 P 0af6e58150ce4b73a4c412dd6c640a60 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0af6e58150ce4b73a4c412dd6c640a60" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 33203 } }
I20260620 14:48:12.327167 8036 raft_consensus.cc:374] T 00000000000000000000000000000000 P 0af6e58150ce4b73a4c412dd6c640a60 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 14:48:12.327287 8036 raft_consensus.cc:729] T 00000000000000000000000000000000 P 0af6e58150ce4b73a4c412dd6c640a60 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0af6e58150ce4b73a4c412dd6c640a60, State: Initialized, Role: FOLLOWER
I20260620 14:48:12.327929 8036 consensus_queue.cc:260] T 00000000000000000000000000000000 P 0af6e58150ce4b73a4c412dd6c640a60 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0af6e58150ce4b73a4c412dd6c640a60" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 33203 } }
I20260620 14:48:12.328176 8036 raft_consensus.cc:388] T 00000000000000000000000000000000 P 0af6e58150ce4b73a4c412dd6c640a60 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 14:48:12.328329 8036 raft_consensus.cc:482] T 00000000000000000000000000000000 P 0af6e58150ce4b73a4c412dd6c640a60 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 14:48:12.328506 8036 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 0af6e58150ce4b73a4c412dd6c640a60 [term 0 FOLLOWER]: Advancing to term 1
I20260620 14:48:12.330644 8036 raft_consensus.cc:504] T 00000000000000000000000000000000 P 0af6e58150ce4b73a4c412dd6c640a60 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0af6e58150ce4b73a4c412dd6c640a60" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 33203 } }
I20260620 14:48:12.331172 8036 leader_election.cc:302] T 00000000000000000000000000000000 P 0af6e58150ce4b73a4c412dd6c640a60 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 0af6e58150ce4b73a4c412dd6c640a60; no voters:
I20260620 14:48:12.331830 8036 leader_election.cc:288] T 00000000000000000000000000000000 P 0af6e58150ce4b73a4c412dd6c640a60 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260620 14:48:12.331915 8041 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 0af6e58150ce4b73a4c412dd6c640a60 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 14:48:12.332638 8041 raft_consensus.cc:686] T 00000000000000000000000000000000 P 0af6e58150ce4b73a4c412dd6c640a60 [term 1 LEADER]: Becoming Leader. State: Replica: 0af6e58150ce4b73a4c412dd6c640a60, State: Running, Role: LEADER
I20260620 14:48:12.333204 8041 consensus_queue.cc:237] T 00000000000000000000000000000000 P 0af6e58150ce4b73a4c412dd6c640a60 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0af6e58150ce4b73a4c412dd6c640a60" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 33203 } }
I20260620 14:48:12.334071 8036 sys_catalog.cc:564] T 00000000000000000000000000000000 P 0af6e58150ce4b73a4c412dd6c640a60 [sys.catalog]: configured and running, proceeding with master startup.
I20260620 14:48:12.336287 8043 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0af6e58150ce4b73a4c412dd6c640a60 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 0af6e58150ce4b73a4c412dd6c640a60. Latest consensus state: current_term: 1 leader_uuid: "0af6e58150ce4b73a4c412dd6c640a60" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0af6e58150ce4b73a4c412dd6c640a60" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 33203 } } }
I20260620 14:48:12.336244 8042 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0af6e58150ce4b73a4c412dd6c640a60 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "0af6e58150ce4b73a4c412dd6c640a60" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0af6e58150ce4b73a4c412dd6c640a60" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 33203 } } }
I20260620 14:48:12.336674 8043 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0af6e58150ce4b73a4c412dd6c640a60 [sys.catalog]: This master's current role is: LEADER
I20260620 14:48:12.336683 8042 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0af6e58150ce4b73a4c412dd6c640a60 [sys.catalog]: This master's current role is: LEADER
I20260620 14:48:12.338404 8048 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 14:48:12.344213 8048 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 14:48:12.353490 8048 catalog_manager.cc:1347] Generated new cluster ID: 0e8bcb46c1b04f17b373212c9d06346f
I20260620 14:48:12.353631 8048 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 14:48:12.364506 8048 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 14:48:12.366380 8048 catalog_manager.cc:1504] Loading token signing keys...
I20260620 14:48:12.381618 8048 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 0af6e58150ce4b73a4c412dd6c640a60: Generated new TSK 0
I20260620 14:48:12.382812 8048 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for oryx/127.0.97.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.0.97.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.0.97.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskUci3DU/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781966829091300-388-0/krb5kdc/oryx_127.0.97.1.keytab.
Entry for principal oryx/127.0.97.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskUci3DU/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781966829091300-388-0/krb5kdc/oryx_127.0.97.1.keytab.
I20260620 14:48:12.433352 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskUci3DU/build/asan/bin/kudu
/tmp/dist-test-taskUci3DU/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskUci3DU/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781966829091300-388-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskUci3DU/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781966829091300-388-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskUci3DU/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781966829091300-388-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskUci3DU/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781966829091300-388-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.1:0
--local_ip_for_outbound_sockets=127.0.97.1
--webserver_interface=127.0.97.1
--webserver_port=0
--tserver_master_addrs=127.0.97.62:33203
--keytab_file=/tmp/dist-test-taskUci3DU/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781966829091300-388-0/krb5kdc/oryx_127.0.97.1.keytab
--principal=oryx/127.0.97.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:43131
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-taskUci3DU/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781966829091300-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskUci3DU/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781966829091300-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskUci3DU/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781966829091300-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 14:48:12.773793 8062 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 14:48:12.774089 8062 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 14:48:12.774219 8062 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 14:48:12.781692 8062 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 14:48:12.781833 8062 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 14:48:12.781982 8062 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260620 14:48:12.782105 8062 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.1
I20260620 14:48:12.791813 8062 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:43131
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskUci3DU/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781966829091300-388-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskUci3DU/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781966829091300-388-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskUci3DU/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781966829091300-388-0/krb5kdc/oryx_127.0.97.1.keytab
--principal=oryx/127.0.97.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskUci3DU/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781966829091300-388-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.97.1
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.0.97.62:33203
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.1
--log_dir=/tmp/dist-test-taskUci3DU/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781966829091300-388-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 13:41:20 UTC on ccycloud.quasar-vdagoi.root.comops.site
build id 4457462
ASAN enabled
I20260620 14:48:12.793156 8062 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 14:48:12.794543 8062 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 14:48:12.807991 8067 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 14:48:12.807942 8070 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 14:48:12.807889 8068 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 14:48:12.808768 8062 server_base.cc:1034] running on GCE node
I20260620 14:48:12.809443 8062 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 14:48:12.810326 8062 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 14:48:12.811538 8062 hybrid_clock.cc:648] HybridClock initialized: now 1781966892811479 us; error 33 us; skew 500 ppm
I20260620 14:48:12.811913 8062 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 14:48:12 dist-test-slave-q9j6 krb5kdc[7946](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781966892, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.97.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 14:48:12.815148 8062 init.cc:377] Logged in from keytab as oryx/127.0.97.1@KRBTEST.COM (short username oryx)
I20260620 14:48:12.816793 8062 webserver.cc:460] Webserver started at http://127.0.97.1:34155/ using document root <none> and password file <none>
I20260620 14:48:12.817626 8062 fs_manager.cc:359] Metadata directory not provided
I20260620 14:48:12.817719 8062 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 14:48:12.818038 8062 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 14:48:12.821503 8062 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskUci3DU/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781966829091300-388-0/minicluster-data/ts-0/data/instance:
uuid: "3a2fba6e705d4e23af93e43abe62f350"
format_stamp: "Formatted at 2026-06-20 14:48:12 on dist-test-slave-q9j6"
server_key: "8c7e2f0666047843a5bddddaca06c3c4"
server_key_iv: "1e8500c9b261dcd6d657056fd8354b5e"
server_key_version: "encryptionkey@0"
I20260620 14:48:12.822276 8062 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskUci3DU/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781966829091300-388-0/minicluster-data/ts-0/wal/instance:
uuid: "3a2fba6e705d4e23af93e43abe62f350"
format_stamp: "Formatted at 2026-06-20 14:48:12 on dist-test-slave-q9j6"
server_key: "8c7e2f0666047843a5bddddaca06c3c4"
server_key_iv: "1e8500c9b261dcd6d657056fd8354b5e"
server_key_version: "encryptionkey@0"
I20260620 14:48:12.827509 8062 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.007s sys 0.000s
I20260620 14:48:12.830957 8077 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 14:48:12.832379 8062 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.002s
I20260620 14:48:12.832558 8062 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskUci3DU/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781966829091300-388-0/minicluster-data/ts-0/data,/tmp/dist-test-taskUci3DU/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781966829091300-388-0/minicluster-data/ts-0/wal
uuid: "3a2fba6e705d4e23af93e43abe62f350"
format_stamp: "Formatted at 2026-06-20 14:48:12 on dist-test-slave-q9j6"
server_key: "8c7e2f0666047843a5bddddaca06c3c4"
server_key_iv: "1e8500c9b261dcd6d657056fd8354b5e"
server_key_version: "encryptionkey@0"
I20260620 14:48:12.832806 8062 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskUci3DU/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781966829091300-388-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskUci3DU/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781966829091300-388-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskUci3DU/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781966829091300-388-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 14:48:12.862783 8062 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 14:48:12.867000 8062 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 14:48:12.867291 8062 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 14:48:12.870065 8062 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 14:48:12.870179 8062 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 14:48:12.870333 8062 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 14:48:12.870410 8062 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Jun 20 14:48:12 dist-test-slave-q9j6 krb5kdc[7946](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781966892, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.97.1@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM
I20260620 14:48:12.895830 8086 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:48:12.870735 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.1:36451 (local address 127.0.97.62:33203)
0620 14:48:12.871337 (+ 602us) server_negotiation.cc:206] Beginning negotiation
0620 14:48:12.871344 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0620 14:48:12.872565 (+ 1221us) server_negotiation.cc:407] Connection header received
0620 14:48:12.874603 (+ 2038us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:48:12.874609 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:48:12.874702 (+ 93us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:48:12.874818 (+ 116us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:48:12.877577 (+ 2759us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:48:12.878566 (+ 989us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:48:12.882160 (+ 3594us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:48:12.882572 (+ 412us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:48:12.882692 (+ 120us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:48:12.888514 (+ 5822us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 14:48:12.888524 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 14:48:12.888529 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 14:48:12.888577 (+ 48us) server_negotiation.cc:922] calling sasl_server_start()
0620 14:48:12.891885 (+ 3308us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:48:12.893072 (+ 1187us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:48:12.893079 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:48:12.893085 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:48:12.893268 (+ 183us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:48:12.893961 (+ 693us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:48:12.893967 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:48:12.893971 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:48:12.894248 (+ 277us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 14:48:12.894406 (+ 158us) server_negotiation.cc:1035] Waiting for connection context
0620 14:48:12.895214 (+ 808us) server_negotiation.cc:299] Negotiation successful
0620 14:48:12.895438 (+ 224us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":478,"thread_start_us":375,"threads_started":1}
I20260620 14:48:12.897027 8087 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:48:12.871420 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:33203 (local address 127.0.97.1:36451)
0620 14:48:12.872367 (+ 947us) negotiation.cc:107] Waiting for socket to connect
0620 14:48:12.872420 (+ 53us) client_negotiation.cc:174] Beginning negotiation
0620 14:48:12.874306 (+ 1886us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 14:48:12.875285 (+ 979us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 14:48:12.875297 (+ 12us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 14:48:12.876361 (+ 1064us) client_negotiation.cc:189] Negotiated authn=SASL
0620 14:48:12.877283 (+ 922us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:48:12.877309 (+ 26us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:48:12.878771 (+ 1462us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:48:12.878779 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:48:12.881965 (+ 3186us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:48:12.881990 (+ 25us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:48:12.882880 (+ 890us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:48:12.882890 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:48:12.883285 (+ 395us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:48:12.884287 (+ 1002us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 14:48:12.884324 (+ 37us) client_negotiation.cc:656] calling sasl_client_start()
0620 14:48:12.888126 (+ 3802us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 14:48:12.892073 (+ 3947us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 14:48:12.892081 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 14:48:12.892099 (+ 18us) client_negotiation.cc:760] calling sasl_client_step()
0620 14:48:12.892858 (+ 759us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 14:48:12.893515 (+ 657us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 14:48:12.893521 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 14:48:12.893527 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0620 14:48:12.893816 (+ 289us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 14:48:12.894379 (+ 563us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 14:48:12.894387 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 14:48:12.894814 (+ 427us) client_negotiation.cc:769] Sending connection context
0620 14:48:12.895127 (+ 313us) client_negotiation.cc:240] Negotiation successful
0620 14:48:12.895719 (+ 592us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":660,"thread_start_us":411,"threads_started":1}
I20260620 14:48:12.917527 8151 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:48:12.902955 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:33203 (local address 127.0.97.1:59813)
0620 14:48:12.903701 (+ 746us) negotiation.cc:107] Waiting for socket to connect
0620 14:48:12.903717 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0620 14:48:12.903884 (+ 167us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 14:48:12.904242 (+ 358us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 14:48:12.904246 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 14:48:12.904710 (+ 464us) client_negotiation.cc:189] Negotiated authn=SASL
0620 14:48:12.904984 (+ 274us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:48:12.904993 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:48:12.905962 (+ 969us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:48:12.905970 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:48:12.906991 (+ 1021us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:48:12.907003 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:48:12.907843 (+ 840us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:48:12.907851 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:48:12.908248 (+ 397us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:48:12.908950 (+ 702us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 14:48:12.908967 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0620 14:48:12.910150 (+ 1183us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 14:48:12.914515 (+ 4365us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 14:48:12.914520 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 14:48:12.914523 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0620 14:48:12.914927 (+ 404us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 14:48:12.915341 (+ 414us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 14:48:12.915344 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 14:48:12.915346 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 14:48:12.915465 (+ 119us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 14:48:12.916158 (+ 693us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 14:48:12.916163 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 14:48:12.916487 (+ 324us) client_negotiation.cc:769] Sending connection context
0620 14:48:12.916690 (+ 203us) client_negotiation.cc:240] Negotiation successful
0620 14:48:12.917072 (+ 382us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":565,"thread_start_us":402,"threads_started":1}
I20260620 14:48:12.917649 8086 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:48:12.902958 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.1:59813 (local address 127.0.97.62:33203)
0620 14:48:12.903186 (+ 228us) server_negotiation.cc:206] Beginning negotiation
0620 14:48:12.903196 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0620 14:48:12.903805 (+ 609us) server_negotiation.cc:407] Connection header received
0620 14:48:12.904029 (+ 224us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:48:12.904035 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:48:12.904116 (+ 81us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:48:12.904230 (+ 114us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:48:12.905125 (+ 895us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:48:12.905820 (+ 695us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:48:12.907141 (+ 1321us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:48:12.907547 (+ 406us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:48:12.907780 (+ 233us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:48:12.910431 (+ 2651us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 14:48:12.910441 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 14:48:12.910468 (+ 27us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 14:48:12.910518 (+ 50us) server_negotiation.cc:922] calling sasl_server_start()
0620 14:48:12.914375 (+ 3857us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:48:12.915060 (+ 685us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:48:12.915066 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:48:12.915069 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:48:12.915229 (+ 160us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:48:12.915592 (+ 363us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:48:12.915620 (+ 28us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:48:12.915632 (+ 12us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:48:12.916043 (+ 411us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 14:48:12.916187 (+ 144us) server_negotiation.cc:1035] Waiting for connection context
0620 14:48:12.917076 (+ 889us) server_negotiation.cc:299] Negotiation successful
0620 14:48:12.917316 (+ 240us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":59}
I20260620 14:48:12.929104 8062 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.1:39761
I20260620 14:48:12.929144 8197 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.1:39761 every 8 connection(s)
I20260620 14:48:12.930379 8062 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskUci3DU/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781966829091300-388-0/minicluster-data/ts-0/data/info.pb
I20260620 14:48:12.932519 8198 heartbeater.cc:347] Connected to a master server at 127.0.97.62:33203
I20260620 14:48:12.932716 8198 heartbeater.cc:464] Registering TS with master...
I20260620 14:48:12.933320 8198 heartbeater.cc:511] Master 127.0.97.62:33203 requested a full tablet report, sending...
I20260620 14:48:12.935415 7991 ts_manager.cc:194] Registered new tserver with Master: 3a2fba6e705d4e23af93e43abe62f350 (127.0.97.1:39761)
I20260620 14:48:12.937044 7991 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.0.97.1@KRBTEST.COM'} at 127.0.97.1:36451
I20260620 14:48:12.938941 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskUci3DU/build/asan/bin/kudu as pid 8062
I20260620 14:48:12.939165 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskUci3DU/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781966829091300-388-0/minicluster-data/ts-0/wal/instance
I20260620 14:48:12.939957 388 external_mini_cluster.cc:1428] Setting key a654052c4c2e52698f97f7f0e02ce9ee
WARNING: no policy specified for oryx/127.0.97.2@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.0.97.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.0.97.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskUci3DU/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781966829091300-388-0/krb5kdc/oryx_127.0.97.2.keytab.
Entry for principal oryx/127.0.97.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskUci3DU/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781966829091300-388-0/krb5kdc/oryx_127.0.97.2.keytab.
I20260620 14:48:12.984926 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskUci3DU/build/asan/bin/kudu
/tmp/dist-test-taskUci3DU/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskUci3DU/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781966829091300-388-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskUci3DU/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781966829091300-388-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskUci3DU/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781966829091300-388-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskUci3DU/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781966829091300-388-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.2:0
--local_ip_for_outbound_sockets=127.0.97.2
--webserver_interface=127.0.97.2
--webserver_port=0
--tserver_master_addrs=127.0.97.62:33203
--keytab_file=/tmp/dist-test-taskUci3DU/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781966829091300-388-0/krb5kdc/oryx_127.0.97.2.keytab
--principal=oryx/127.0.97.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:43131
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-taskUci3DU/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781966829091300-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskUci3DU/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781966829091300-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskUci3DU/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781966829091300-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 14:48:13.319159 8202 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 14:48:13.319442 8202 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 14:48:13.319644 8202 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 14:48:13.326973 8202 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 14:48:13.327085 8202 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 14:48:13.327229 8202 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260620 14:48:13.327361 8202 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.2
I20260620 14:48:13.337661 8202 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:43131
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskUci3DU/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781966829091300-388-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskUci3DU/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781966829091300-388-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskUci3DU/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781966829091300-388-0/krb5kdc/oryx_127.0.97.2.keytab
--principal=oryx/127.0.97.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskUci3DU/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781966829091300-388-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.97.2
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.0.97.62:33203
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.2
--log_dir=/tmp/dist-test-taskUci3DU/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781966829091300-388-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 13:41:20 UTC on ccycloud.quasar-vdagoi.root.comops.site
build id 4457462
ASAN enabled
I20260620 14:48:13.339027 8202 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 14:48:13.340482 8202 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 14:48:13.353861 8207 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 14:48:13.354004 8208 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 14:48:13.354112 8210 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 14:48:13.355100 8202 server_base.cc:1034] running on GCE node
I20260620 14:48:13.355844 8202 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 14:48:13.356827 8202 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 14:48:13.358085 8202 hybrid_clock.cc:648] HybridClock initialized: now 1781966893358021 us; error 44 us; skew 500 ppm
I20260620 14:48:13.358402 8202 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 14:48:13 dist-test-slave-q9j6 krb5kdc[7946](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781966893, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.97.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 14:48:13.361671 8202 init.cc:377] Logged in from keytab as oryx/127.0.97.2@KRBTEST.COM (short username oryx)
I20260620 14:48:13.363216 8202 webserver.cc:460] Webserver started at http://127.0.97.2:38403/ using document root <none> and password file <none>
I20260620 14:48:13.364028 8202 fs_manager.cc:359] Metadata directory not provided
I20260620 14:48:13.364161 8202 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 14:48:13.364496 8202 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 14:48:13.367517 8202 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskUci3DU/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781966829091300-388-0/minicluster-data/ts-1/data/instance:
uuid: "a99fedfd9f1844aeb7e6362e68c7d072"
format_stamp: "Formatted at 2026-06-20 14:48:13 on dist-test-slave-q9j6"
server_key: "9b3da57d2b7ebe38d0878b8ffb61a575"
server_key_iv: "b969c9a1ed8edf48f2ed227ab8b8fecd"
server_key_version: "encryptionkey@0"
I20260620 14:48:13.368623 8202 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskUci3DU/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781966829091300-388-0/minicluster-data/ts-1/wal/instance:
uuid: "a99fedfd9f1844aeb7e6362e68c7d072"
format_stamp: "Formatted at 2026-06-20 14:48:13 on dist-test-slave-q9j6"
server_key: "9b3da57d2b7ebe38d0878b8ffb61a575"
server_key_iv: "b969c9a1ed8edf48f2ed227ab8b8fecd"
server_key_version: "encryptionkey@0"
I20260620 14:48:13.373886 8202 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260620 14:48:13.377367 8217 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 14:48:13.378834 8202 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260620 14:48:13.379043 8202 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskUci3DU/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781966829091300-388-0/minicluster-data/ts-1/data,/tmp/dist-test-taskUci3DU/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781966829091300-388-0/minicluster-data/ts-1/wal
uuid: "a99fedfd9f1844aeb7e6362e68c7d072"
format_stamp: "Formatted at 2026-06-20 14:48:13 on dist-test-slave-q9j6"
server_key: "9b3da57d2b7ebe38d0878b8ffb61a575"
server_key_iv: "b969c9a1ed8edf48f2ed227ab8b8fecd"
server_key_version: "encryptionkey@0"
I20260620 14:48:13.379312 8202 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskUci3DU/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781966829091300-388-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskUci3DU/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781966829091300-388-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskUci3DU/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781966829091300-388-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 14:48:13.410683 8202 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 14:48:13.415354 8202 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 14:48:13.415724 8202 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 14:48:13.418607 8202 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 14:48:13.418696 8202 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 14:48:13.418855 8202 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 14:48:13.418923 8202 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Jun 20 14:48:13 dist-test-slave-q9j6 krb5kdc[7946](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781966893, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.97.2@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM
I20260620 14:48:13.448261 8227 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:48:13.420251 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:33203 (local address 127.0.97.2:44451)
0620 14:48:13.421116 (+ 865us) negotiation.cc:107] Waiting for socket to connect
0620 14:48:13.421166 (+ 50us) client_negotiation.cc:174] Beginning negotiation
0620 14:48:13.423386 (+ 2220us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 14:48:13.424696 (+ 1310us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 14:48:13.424716 (+ 20us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 14:48:13.425763 (+ 1047us) client_negotiation.cc:189] Negotiated authn=SASL
0620 14:48:13.426745 (+ 982us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:48:13.426782 (+ 37us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:48:13.428575 (+ 1793us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:48:13.428587 (+ 12us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:48:13.431710 (+ 3123us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:48:13.431748 (+ 38us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:48:13.432942 (+ 1194us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:48:13.432952 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:48:13.433421 (+ 469us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:48:13.434454 (+ 1033us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 14:48:13.434492 (+ 38us) client_negotiation.cc:656] calling sasl_client_start()
0620 14:48:13.438785 (+ 4293us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 14:48:13.443002 (+ 4217us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 14:48:13.443012 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 14:48:13.443033 (+ 21us) client_negotiation.cc:760] calling sasl_client_step()
0620 14:48:13.443511 (+ 478us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 14:48:13.443963 (+ 452us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 14:48:13.443969 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 14:48:13.443975 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0620 14:48:13.444331 (+ 356us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 14:48:13.445023 (+ 692us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 14:48:13.445031 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 14:48:13.445618 (+ 587us) client_negotiation.cc:769] Sending connection context
0620 14:48:13.446017 (+ 399us) client_negotiation.cc:240] Negotiation successful
0620 14:48:13.446689 (+ 672us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":575,"thread_start_us":373,"threads_started":1}
I20260620 14:48:13.448843 8226 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:48:13.419475 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.2:44451 (local address 127.0.97.62:33203)
0620 14:48:13.419986 (+ 511us) server_negotiation.cc:206] Beginning negotiation
0620 14:48:13.419991 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 14:48:13.421296 (+ 1305us) server_negotiation.cc:407] Connection header received
0620 14:48:13.423816 (+ 2520us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:48:13.423823 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:48:13.423938 (+ 115us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:48:13.424044 (+ 106us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:48:13.427067 (+ 3023us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:48:13.428290 (+ 1223us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:48:13.432106 (+ 3816us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:48:13.432565 (+ 459us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:48:13.432753 (+ 188us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:48:13.439082 (+ 6329us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 14:48:13.439087 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 14:48:13.439091 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 14:48:13.439129 (+ 38us) server_negotiation.cc:922] calling sasl_server_start()
0620 14:48:13.442833 (+ 3704us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:48:13.443704 (+ 871us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:48:13.443710 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:48:13.443714 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:48:13.443842 (+ 128us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:48:13.444474 (+ 632us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:48:13.444480 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:48:13.444483 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:48:13.444890 (+ 407us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 14:48:13.445039 (+ 149us) server_negotiation.cc:1035] Waiting for connection context
0620 14:48:13.447942 (+ 2903us) server_negotiation.cc:299] Negotiation successful
0620 14:48:13.448269 (+ 327us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":407,"thread_start_us":303,"threads_started":1}
I20260620 14:48:13.472044 8226 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:48:13.455152 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.2:43873 (local address 127.0.97.62:33203)
0620 14:48:13.455367 (+ 215us) server_negotiation.cc:206] Beginning negotiation
0620 14:48:13.455375 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0620 14:48:13.456348 (+ 973us) server_negotiation.cc:407] Connection header received
0620 14:48:13.456553 (+ 205us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:48:13.456557 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:48:13.456622 (+ 65us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:48:13.456753 (+ 131us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:48:13.458677 (+ 1924us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:48:13.459640 (+ 963us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:48:13.462226 (+ 2586us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:48:13.462652 (+ 426us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:48:13.462816 (+ 164us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:48:13.465340 (+ 2524us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 14:48:13.465351 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 14:48:13.465359 (+ 8us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 14:48:13.465413 (+ 54us) server_negotiation.cc:922] calling sasl_server_start()
0620 14:48:13.469214 (+ 3801us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:48:13.469913 (+ 699us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:48:13.469921 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:48:13.469927 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:48:13.470104 (+ 177us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:48:13.470517 (+ 413us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:48:13.470524 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:48:13.470527 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:48:13.470861 (+ 334us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 14:48:13.471021 (+ 160us) server_negotiation.cc:1035] Waiting for connection context
0620 14:48:13.471416 (+ 395us) server_negotiation.cc:299] Negotiation successful
0620 14:48:13.471696 (+ 280us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":70}
I20260620 14:48:13.472242 8297 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:48:13.455135 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:33203 (local address 127.0.97.2:43873)
0620 14:48:13.456208 (+ 1073us) negotiation.cc:107] Waiting for socket to connect
0620 14:48:13.456239 (+ 31us) client_negotiation.cc:174] Beginning negotiation
0620 14:48:13.456426 (+ 187us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 14:48:13.456779 (+ 353us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 14:48:13.456787 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 14:48:13.457634 (+ 847us) client_negotiation.cc:189] Negotiated authn=SASL
0620 14:48:13.458465 (+ 831us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:48:13.458486 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:48:13.459836 (+ 1350us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:48:13.459845 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:48:13.462050 (+ 2205us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:48:13.462066 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:48:13.462889 (+ 823us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:48:13.462901 (+ 12us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:48:13.463055 (+ 154us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:48:13.463858 (+ 803us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 14:48:13.463914 (+ 56us) client_negotiation.cc:656] calling sasl_client_start()
0620 14:48:13.465077 (+ 1163us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 14:48:13.469367 (+ 4290us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 14:48:13.469373 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 14:48:13.469377 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0620 14:48:13.469777 (+ 400us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 14:48:13.470259 (+ 482us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 14:48:13.470262 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 14:48:13.470265 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0620 14:48:13.470395 (+ 130us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 14:48:13.470986 (+ 591us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 14:48:13.470991 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 14:48:13.471169 (+ 178us) client_negotiation.cc:769] Sending connection context
0620 14:48:13.471327 (+ 158us) client_negotiation.cc:240] Negotiation successful
0620 14:48:13.471729 (+ 402us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":848,"thread_start_us":688,"threads_started":1}
I20260620 14:48:13.480530 8202 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.2:43441
I20260620 14:48:13.480557 8337 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.2:43441 every 8 connection(s)
I20260620 14:48:13.481786 8202 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskUci3DU/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781966829091300-388-0/minicluster-data/ts-1/data/info.pb
I20260620 14:48:13.484020 8338 heartbeater.cc:347] Connected to a master server at 127.0.97.62:33203
I20260620 14:48:13.484272 8338 heartbeater.cc:464] Registering TS with master...
I20260620 14:48:13.484925 8338 heartbeater.cc:511] Master 127.0.97.62:33203 requested a full tablet report, sending...
I20260620 14:48:13.486260 7991 ts_manager.cc:194] Registered new tserver with Master: a99fedfd9f1844aeb7e6362e68c7d072 (127.0.97.2:43441)
I20260620 14:48:13.487099 7991 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.0.97.2@KRBTEST.COM'} at 127.0.97.2:44451
I20260620 14:48:13.488655 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskUci3DU/build/asan/bin/kudu as pid 8202
I20260620 14:48:13.488862 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskUci3DU/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781966829091300-388-0/minicluster-data/ts-1/wal/instance
I20260620 14:48:13.489262 388 external_mini_cluster.cc:1428] Setting key b1178f5701549412faada1a5d14b8f5f
WARNING: no policy specified for oryx/127.0.97.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.0.97.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.0.97.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskUci3DU/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781966829091300-388-0/krb5kdc/oryx_127.0.97.3.keytab.
Entry for principal oryx/127.0.97.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskUci3DU/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781966829091300-388-0/krb5kdc/oryx_127.0.97.3.keytab.
I20260620 14:48:13.532660 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskUci3DU/build/asan/bin/kudu
/tmp/dist-test-taskUci3DU/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskUci3DU/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781966829091300-388-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskUci3DU/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781966829091300-388-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskUci3DU/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781966829091300-388-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskUci3DU/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781966829091300-388-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.3:0
--local_ip_for_outbound_sockets=127.0.97.3
--webserver_interface=127.0.97.3
--webserver_port=0
--tserver_master_addrs=127.0.97.62:33203
--keytab_file=/tmp/dist-test-taskUci3DU/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781966829091300-388-0/krb5kdc/oryx_127.0.97.3.keytab
--principal=oryx/127.0.97.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:43131
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-taskUci3DU/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781966829091300-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskUci3DU/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781966829091300-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskUci3DU/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781966829091300-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 14:48:13.861474 8342 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 14:48:13.861753 8342 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 14:48:13.861941 8342 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 14:48:13.869328 8342 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 14:48:13.869457 8342 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 14:48:13.869606 8342 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260620 14:48:13.869741 8342 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.3
I20260620 14:48:13.879695 8342 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:43131
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskUci3DU/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781966829091300-388-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskUci3DU/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781966829091300-388-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskUci3DU/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781966829091300-388-0/krb5kdc/oryx_127.0.97.3.keytab
--principal=oryx/127.0.97.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskUci3DU/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781966829091300-388-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.97.3
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.0.97.62:33203
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.3
--log_dir=/tmp/dist-test-taskUci3DU/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781966829091300-388-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 13:41:20 UTC on ccycloud.quasar-vdagoi.root.comops.site
build id 4457462
ASAN enabled
I20260620 14:48:13.881037 8342 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 14:48:13.882483 8342 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 14:48:13.895592 8350 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 14:48:13.895843 8347 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 14:48:13.895759 8348 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 14:48:13.896891 8342 server_base.cc:1034] running on GCE node
I20260620 14:48:13.897444 8342 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 14:48:13.898283 8342 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 14:48:13.899477 8342 hybrid_clock.cc:648] HybridClock initialized: now 1781966893899410 us; error 42 us; skew 500 ppm
I20260620 14:48:13.899797 8342 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 14:48:13 dist-test-slave-q9j6 krb5kdc[7946](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781966893, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.97.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 14:48:13.902794 8342 init.cc:377] Logged in from keytab as oryx/127.0.97.3@KRBTEST.COM (short username oryx)
I20260620 14:48:13.904186 8342 webserver.cc:460] Webserver started at http://127.0.97.3:45541/ using document root <none> and password file <none>
I20260620 14:48:13.904829 8342 fs_manager.cc:359] Metadata directory not provided
I20260620 14:48:13.904913 8342 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 14:48:13.905166 8342 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 14:48:13.908042 8342 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskUci3DU/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781966829091300-388-0/minicluster-data/ts-2/data/instance:
uuid: "1a858e39015649089fd200123f040125"
format_stamp: "Formatted at 2026-06-20 14:48:13 on dist-test-slave-q9j6"
server_key: "5f2c0088cf4d6f34900ef3e529ebf238"
server_key_iv: "c569427e62fe46629ae5d8f61a914f8d"
server_key_version: "encryptionkey@0"
I20260620 14:48:13.908795 8342 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskUci3DU/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781966829091300-388-0/minicluster-data/ts-2/wal/instance:
uuid: "1a858e39015649089fd200123f040125"
format_stamp: "Formatted at 2026-06-20 14:48:13 on dist-test-slave-q9j6"
server_key: "5f2c0088cf4d6f34900ef3e529ebf238"
server_key_iv: "c569427e62fe46629ae5d8f61a914f8d"
server_key_version: "encryptionkey@0"
I20260620 14:48:13.913869 8342 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260620 14:48:13.917330 8357 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 14:48:13.918780 8342 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260620 14:48:13.918916 8342 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskUci3DU/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781966829091300-388-0/minicluster-data/ts-2/data,/tmp/dist-test-taskUci3DU/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781966829091300-388-0/minicluster-data/ts-2/wal
uuid: "1a858e39015649089fd200123f040125"
format_stamp: "Formatted at 2026-06-20 14:48:13 on dist-test-slave-q9j6"
server_key: "5f2c0088cf4d6f34900ef3e529ebf238"
server_key_iv: "c569427e62fe46629ae5d8f61a914f8d"
server_key_version: "encryptionkey@0"
I20260620 14:48:13.919098 8342 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskUci3DU/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781966829091300-388-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskUci3DU/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781966829091300-388-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskUci3DU/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781966829091300-388-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 14:48:13.938246 8342 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 14:48:13.940497 8198 heartbeater.cc:503] Master 127.0.97.62:33203 was elected leader, sending a full tablet report...
I20260620 14:48:13.942818 8342 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 14:48:13.943092 8342 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 14:48:13.945792 8342 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 14:48:13.945910 8342 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 14:48:13.946069 8342 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 14:48:13.946152 8342 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Jun 20 14:48:13 dist-test-slave-q9j6 krb5kdc[7946](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781966893, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.97.3@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM
I20260620 14:48:13.972707 8226 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:48:13.946265 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.3:46461 (local address 127.0.97.62:33203)
0620 14:48:13.946505 (+ 240us) server_negotiation.cc:206] Beginning negotiation
0620 14:48:13.946515 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0620 14:48:13.948067 (+ 1552us) server_negotiation.cc:407] Connection header received
0620 14:48:13.951182 (+ 3115us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:48:13.951222 (+ 40us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:48:13.951331 (+ 109us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:48:13.951414 (+ 83us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:48:13.954220 (+ 2806us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:48:13.955582 (+ 1362us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:48:13.957594 (+ 2012us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:48:13.958346 (+ 752us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:48:13.958512 (+ 166us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:48:13.963656 (+ 5144us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 14:48:13.963664 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 14:48:13.963675 (+ 11us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 14:48:13.963712 (+ 37us) server_negotiation.cc:922] calling sasl_server_start()
0620 14:48:13.967443 (+ 3731us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:48:13.968946 (+ 1503us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:48:13.968954 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:48:13.968961 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:48:13.969121 (+ 160us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:48:13.969801 (+ 680us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:48:13.969806 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:48:13.969811 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:48:13.970153 (+ 342us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 14:48:13.970327 (+ 174us) server_negotiation.cc:1035] Waiting for connection context
0620 14:48:13.971878 (+ 1551us) server_negotiation.cc:299] Negotiation successful
0620 14:48:13.972190 (+ 312us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":64}
I20260620 14:48:13.974171 8366 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:48:13.946960 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:33203 (local address 127.0.97.3:46461)
0620 14:48:13.947865 (+ 905us) negotiation.cc:107] Waiting for socket to connect
0620 14:48:13.947930 (+ 65us) client_negotiation.cc:174] Beginning negotiation
0620 14:48:13.950718 (+ 2788us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 14:48:13.952131 (+ 1413us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 14:48:13.952146 (+ 15us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 14:48:13.953327 (+ 1181us) client_negotiation.cc:189] Negotiated authn=SASL
0620 14:48:13.954010 (+ 683us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:48:13.954029 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:48:13.955802 (+ 1773us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:48:13.955809 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:48:13.957427 (+ 1618us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:48:13.957440 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:48:13.958635 (+ 1195us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:48:13.958644 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:48:13.958908 (+ 264us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:48:13.959858 (+ 950us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 14:48:13.959902 (+ 44us) client_negotiation.cc:656] calling sasl_client_start()
0620 14:48:13.963275 (+ 3373us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 14:48:13.967679 (+ 4404us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 14:48:13.967696 (+ 17us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 14:48:13.967727 (+ 31us) client_negotiation.cc:760] calling sasl_client_step()
0620 14:48:13.968764 (+ 1037us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 14:48:13.969248 (+ 484us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 14:48:13.969255 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 14:48:13.969261 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0620 14:48:13.969623 (+ 362us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 14:48:13.970417 (+ 794us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 14:48:13.970432 (+ 15us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 14:48:13.971080 (+ 648us) client_negotiation.cc:769] Sending connection context
0620 14:48:13.971508 (+ 428us) client_negotiation.cc:240] Negotiation successful
0620 14:48:13.972245 (+ 737us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":560,"thread_start_us":327,"threads_started":1}
I20260620 14:48:13.990072 8476 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.3:44163 every 8 connection(s)
I20260620 14:48:13.990068 8342 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.3:44163
I20260620 14:48:13.991535 8342 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskUci3DU/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781966829091300-388-0/minicluster-data/ts-2/data/info.pb
I20260620 14:48:13.994169 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskUci3DU/build/asan/bin/kudu as pid 8342
I20260620 14:48:13.994374 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskUci3DU/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781966829091300-388-0/minicluster-data/ts-2/wal/instance
I20260620 14:48:13.994514 8477 heartbeater.cc:347] Connected to a master server at 127.0.97.62:33203
I20260620 14:48:13.994750 8477 heartbeater.cc:464] Registering TS with master...
I20260620 14:48:13.994917 388 external_mini_cluster.cc:1428] Setting key 75062aa2e567451eba24d9cf03c1d812
I20260620 14:48:13.995644 8477 heartbeater.cc:511] Master 127.0.97.62:33203 requested a full tablet report, sending...
I20260620 14:48:13.996652 8226 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:48:13.981122 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.3:42657 (local address 127.0.97.62:33203)
0620 14:48:13.981346 (+ 224us) server_negotiation.cc:206] Beginning negotiation
0620 14:48:13.981355 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0620 14:48:13.982055 (+ 700us) server_negotiation.cc:407] Connection header received
0620 14:48:13.982358 (+ 303us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:48:13.982366 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:48:13.982473 (+ 107us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:48:13.982598 (+ 125us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:48:13.983735 (+ 1137us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:48:13.984730 (+ 995us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:48:13.987493 (+ 2763us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:48:13.988003 (+ 510us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:48:13.988104 (+ 101us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:48:13.990694 (+ 2590us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 14:48:13.990698 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 14:48:13.990701 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 14:48:13.990734 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0620 14:48:13.993631 (+ 2897us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:48:13.994479 (+ 848us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:48:13.994485 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:48:13.994493 (+ 8us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:48:13.994651 (+ 158us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:48:13.995090 (+ 439us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:48:13.995099 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:48:13.995101 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:48:13.995358 (+ 257us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 14:48:13.995514 (+ 156us) server_negotiation.cc:1035] Waiting for connection context
0620 14:48:13.996083 (+ 569us) server_negotiation.cc:299] Negotiation successful
0620 14:48:13.996302 (+ 219us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":68}
I20260620 14:48:13.996742 8455 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:48:13.981167 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:33203 (local address 127.0.97.3:42657)
0620 14:48:13.981939 (+ 772us) negotiation.cc:107] Waiting for socket to connect
0620 14:48:13.981963 (+ 24us) client_negotiation.cc:174] Beginning negotiation
0620 14:48:13.982191 (+ 228us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 14:48:13.982661 (+ 470us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 14:48:13.982670 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 14:48:13.983248 (+ 578us) client_negotiation.cc:189] Negotiated authn=SASL
0620 14:48:13.983573 (+ 325us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:48:13.983587 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:48:13.984920 (+ 1333us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:48:13.984927 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:48:13.987300 (+ 2373us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:48:13.987326 (+ 26us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:48:13.988196 (+ 870us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:48:13.988202 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:48:13.988349 (+ 147us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:48:13.989095 (+ 746us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 14:48:13.989128 (+ 33us) client_negotiation.cc:656] calling sasl_client_start()
0620 14:48:13.990500 (+ 1372us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 14:48:13.993774 (+ 3274us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 14:48:13.993779 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 14:48:13.993783 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0620 14:48:13.994347 (+ 564us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 14:48:13.994779 (+ 432us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 14:48:13.994785 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 14:48:13.994789 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0620 14:48:13.994940 (+ 151us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 14:48:13.995580 (+ 640us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 14:48:13.995590 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 14:48:13.995839 (+ 249us) client_negotiation.cc:769] Sending connection context
0620 14:48:13.995983 (+ 144us) client_negotiation.cc:240] Negotiation successful
0620 14:48:13.996306 (+ 323us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":620,"thread_start_us":466,"threads_started":1}
I20260620 14:48:13.997308 7991 ts_manager.cc:194] Registered new tserver with Master: 1a858e39015649089fd200123f040125 (127.0.97.3:44163)
I20260620 14:48:13.998111 7991 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.0.97.3@KRBTEST.COM'} at 127.0.97.3:46461
I20260620 14:48:13.998288 388 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
Jun 20 14:48:14 dist-test-slave-q9j6 krb5kdc[7946](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781966894, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-user@KRBTEST.COM:
Jun 20 14:48:14 dist-test-slave-q9j6 krb5kdc[7946](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, test-user@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
Jun 20 14:48:14 dist-test-slave-q9j6 krb5kdc[7946](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, test-user@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
W20260620 14:48:14.049090 8226 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 14:48:14.038634 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52202 (local address 127.0.97.62:33203)
0620 14:48:14.038887 (+ 253us) server_negotiation.cc:206] Beginning negotiation
0620 14:48:14.038894 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0620 14:48:14.039078 (+ 184us) server_negotiation.cc:407] Connection header received
0620 14:48:14.039230 (+ 152us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:48:14.039234 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:48:14.039299 (+ 65us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:48:14.039388 (+ 89us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:48:14.040821 (+ 1433us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:48:14.041533 (+ 712us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:48:14.042766 (+ 1233us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:48:14.043198 (+ 432us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:48:14.043296 (+ 98us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:48:14.048745 (+ 5449us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:52202: BlockingRecv error: recv got EOF from 127.0.0.1:52202 (error 108)
Metrics: {"server-negotiator.queue_time_us":85}
Jun 20 14:48:14 dist-test-slave-q9j6 krb5kdc[7946](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781966894, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM
I20260620 14:48:14.065585 8226 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:48:14.052413 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52204 (local address 127.0.97.62:33203)
0620 14:48:14.052580 (+ 167us) server_negotiation.cc:206] Beginning negotiation
0620 14:48:14.052584 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 14:48:14.052909 (+ 325us) server_negotiation.cc:407] Connection header received
0620 14:48:14.053078 (+ 169us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:48:14.053082 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:48:14.053146 (+ 64us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:48:14.053233 (+ 87us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:48:14.054338 (+ 1105us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:48:14.055092 (+ 754us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:48:14.056435 (+ 1343us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:48:14.056845 (+ 410us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:48:14.056948 (+ 103us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:48:14.060788 (+ 3840us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 14:48:14.060793 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 14:48:14.060796 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 14:48:14.060862 (+ 66us) server_negotiation.cc:922] calling sasl_server_start()
0620 14:48:14.063244 (+ 2382us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:48:14.063958 (+ 714us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:48:14.063962 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:48:14.063964 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:48:14.064094 (+ 130us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:48:14.064407 (+ 313us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:48:14.064410 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:48:14.064412 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:48:14.064661 (+ 249us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 14:48:14.064856 (+ 195us) server_negotiation.cc:1035] Waiting for connection context
0620 14:48:14.065184 (+ 328us) server_negotiation.cc:299] Negotiation successful
0620 14:48:14.065348 (+ 164us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":51}
I20260620 14:48:14.069638 7989 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:52204:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260620 14:48:14.076613 7989 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260620 14:48:14.102326 8496 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:48:14.088697 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36912 (local address 127.0.97.2:43441)
0620 14:48:14.090477 (+ 1780us) server_negotiation.cc:206] Beginning negotiation
0620 14:48:14.090487 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0620 14:48:14.091185 (+ 698us) server_negotiation.cc:407] Connection header received
0620 14:48:14.091683 (+ 498us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:48:14.091695 (+ 12us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:48:14.092006 (+ 311us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:48:14.092187 (+ 181us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 14:48:14.093286 (+ 1099us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:48:14.095053 (+ 1767us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:48:14.098551 (+ 3498us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:48:14.100423 (+ 1872us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:48:14.100552 (+ 129us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:48:14.100834 (+ 282us) server_negotiation.cc:1035] Waiting for connection context
0620 14:48:14.101707 (+ 873us) server_negotiation.cc:299] Negotiation successful
0620 14:48:14.101938 (+ 231us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":566,"thread_start_us":405,"threads_started":1}
I20260620 14:48:14.103523 8495 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:48:14.088533 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.2:43441 (local address 127.0.0.1:36912)
0620 14:48:14.091044 (+ 2511us) negotiation.cc:107] Waiting for socket to connect
0620 14:48:14.091088 (+ 44us) client_negotiation.cc:174] Beginning negotiation
0620 14:48:14.091369 (+ 281us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 14:48:14.092517 (+ 1148us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 14:48:14.092528 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 14:48:14.092568 (+ 40us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 14:48:14.093076 (+ 508us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:48:14.093091 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:48:14.095273 (+ 2182us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:48:14.095281 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:48:14.098325 (+ 3044us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:48:14.098347 (+ 22us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:48:14.101048 (+ 2701us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:48:14.101057 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:48:14.101361 (+ 304us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:48:14.101431 (+ 70us) client_negotiation.cc:769] Sending connection context
0620 14:48:14.102764 (+ 1333us) client_negotiation.cc:240] Negotiation successful
0620 14:48:14.103052 (+ 288us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":2252,"thread_start_us":2115,"threads_started":1}
I20260620 14:48:14.108969 8268 tablet_service.cc:1467] Processing CreateTablet for tablet 6a36724722064849948c8e1c8b904d24 (DEFAULT_TABLE table=test-table [id=ee5ba9f2145b42099311eb4507b636d4]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 14:48:14.111474 8268 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6a36724722064849948c8e1c8b904d24. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 14:48:14.114843 8495 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:48:14.088702 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.3:44163 (local address 127.0.0.1:33502)
0620 14:48:14.104075 (+ 15373us) negotiation.cc:107] Waiting for socket to connect
0620 14:48:14.104099 (+ 24us) client_negotiation.cc:174] Beginning negotiation
0620 14:48:14.104243 (+ 144us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 14:48:14.107591 (+ 3348us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 14:48:14.107622 (+ 31us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 14:48:14.107676 (+ 54us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 14:48:14.107927 (+ 251us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:48:14.107939 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:48:14.110265 (+ 2326us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:48:14.110271 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:48:14.112191 (+ 1920us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:48:14.112208 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:48:14.114035 (+ 1827us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:48:14.114044 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:48:14.114249 (+ 205us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:48:14.114297 (+ 48us) client_negotiation.cc:769] Sending connection context
0620 14:48:14.114390 (+ 93us) client_negotiation.cc:240] Negotiation successful
0620 14:48:14.114543 (+ 153us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":15236,"thread_start_us":19075,"threads_started":1}
I20260620 14:48:14.115015 8498 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:48:14.089105 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33502 (local address 127.0.97.3:44163)
0620 14:48:14.106752 (+ 17647us) server_negotiation.cc:206] Beginning negotiation
0620 14:48:14.106765 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0620 14:48:14.106807 (+ 42us) server_negotiation.cc:407] Connection header received
0620 14:48:14.106915 (+ 108us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:48:14.106924 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:48:14.107249 (+ 325us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:48:14.108132 (+ 883us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 14:48:14.108456 (+ 324us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:48:14.110078 (+ 1622us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:48:14.112386 (+ 2308us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:48:14.113848 (+ 1462us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:48:14.113987 (+ 139us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:48:14.114224 (+ 237us) server_negotiation.cc:1035] Waiting for connection context
0620 14:48:14.114488 (+ 264us) server_negotiation.cc:299] Negotiation successful
0620 14:48:14.114661 (+ 173us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":17424,"thread_start_us":17235,"threads_started":1}
I20260620 14:48:14.116420 8500 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:48:14.090938 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:39761 (local address 127.0.0.1:56560)
0620 14:48:14.105728 (+ 14790us) negotiation.cc:107] Waiting for socket to connect
0620 14:48:14.105749 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0620 14:48:14.105871 (+ 122us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 14:48:14.108114 (+ 2243us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 14:48:14.108119 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 14:48:14.108157 (+ 38us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 14:48:14.109372 (+ 1215us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:48:14.109386 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:48:14.112008 (+ 2622us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:48:14.112019 (+ 11us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:48:14.113472 (+ 1453us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:48:14.113485 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:48:14.115571 (+ 2086us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:48:14.115575 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:48:14.115852 (+ 277us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:48:14.115888 (+ 36us) client_negotiation.cc:769] Sending connection context
0620 14:48:14.115975 (+ 87us) client_negotiation.cc:240] Negotiation successful
0620 14:48:14.116125 (+ 150us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":14636,"thread_start_us":15030,"threads_started":1}
I20260620 14:48:14.117259 8499 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:48:14.091114 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56560 (local address 127.0.97.1:39761)
0620 14:48:14.107089 (+ 15975us) server_negotiation.cc:206] Beginning negotiation
0620 14:48:14.107098 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0620 14:48:14.107126 (+ 28us) server_negotiation.cc:407] Connection header received
0620 14:48:14.107215 (+ 89us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:48:14.107221 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:48:14.107449 (+ 228us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:48:14.108105 (+ 656us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 14:48:14.109667 (+ 1562us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:48:14.111756 (+ 2089us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:48:14.113659 (+ 1903us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:48:14.115082 (+ 1423us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:48:14.115210 (+ 128us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:48:14.115433 (+ 223us) server_negotiation.cc:1035] Waiting for connection context
0620 14:48:14.116793 (+ 1360us) server_negotiation.cc:299] Negotiation successful
0620 14:48:14.116972 (+ 179us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":15826,"thread_start_us":15690,"threads_started":1}
I20260620 14:48:14.119372 8407 tablet_service.cc:1467] Processing CreateTablet for tablet 6a36724722064849948c8e1c8b904d24 (DEFAULT_TABLE table=test-table [id=ee5ba9f2145b42099311eb4507b636d4]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 14:48:14.120795 8407 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6a36724722064849948c8e1c8b904d24. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 14:48:14.121095 8128 tablet_service.cc:1467] Processing CreateTablet for tablet 6a36724722064849948c8e1c8b904d24 (DEFAULT_TABLE table=test-table [id=ee5ba9f2145b42099311eb4507b636d4]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 14:48:14.122529 8128 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6a36724722064849948c8e1c8b904d24. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 14:48:14.127985 8501 tablet_bootstrap.cc:492] T 6a36724722064849948c8e1c8b904d24 P a99fedfd9f1844aeb7e6362e68c7d072: Bootstrap starting.
I20260620 14:48:14.131968 8501 tablet_bootstrap.cc:654] T 6a36724722064849948c8e1c8b904d24 P a99fedfd9f1844aeb7e6362e68c7d072: Neither blocks nor log segments found. Creating new log.
I20260620 14:48:14.133461 8501 log.cc:824] T 6a36724722064849948c8e1c8b904d24 P a99fedfd9f1844aeb7e6362e68c7d072: Log is configured to *not* fsync() on all Append() calls
I20260620 14:48:14.134243 8502 tablet_bootstrap.cc:492] T 6a36724722064849948c8e1c8b904d24 P 1a858e39015649089fd200123f040125: Bootstrap starting.
I20260620 14:48:14.138110 8502 tablet_bootstrap.cc:654] T 6a36724722064849948c8e1c8b904d24 P 1a858e39015649089fd200123f040125: Neither blocks nor log segments found. Creating new log.
I20260620 14:48:14.138453 8501 tablet_bootstrap.cc:492] T 6a36724722064849948c8e1c8b904d24 P a99fedfd9f1844aeb7e6362e68c7d072: No bootstrap required, opened a new log
I20260620 14:48:14.138736 8501 ts_tablet_manager.cc:1397] T 6a36724722064849948c8e1c8b904d24 P a99fedfd9f1844aeb7e6362e68c7d072: Time spent bootstrapping tablet: real 0.011s user 0.010s sys 0.000s
I20260620 14:48:14.138733 8504 tablet_bootstrap.cc:492] T 6a36724722064849948c8e1c8b904d24 P 3a2fba6e705d4e23af93e43abe62f350: Bootstrap starting.
I20260620 14:48:14.139487 8502 log.cc:824] T 6a36724722064849948c8e1c8b904d24 P 1a858e39015649089fd200123f040125: Log is configured to *not* fsync() on all Append() calls
I20260620 14:48:14.142307 8504 tablet_bootstrap.cc:654] T 6a36724722064849948c8e1c8b904d24 P 3a2fba6e705d4e23af93e43abe62f350: Neither blocks nor log segments found. Creating new log.
I20260620 14:48:14.143728 8504 log.cc:824] T 6a36724722064849948c8e1c8b904d24 P 3a2fba6e705d4e23af93e43abe62f350: Log is configured to *not* fsync() on all Append() calls
I20260620 14:48:14.143352 8502 tablet_bootstrap.cc:492] T 6a36724722064849948c8e1c8b904d24 P 1a858e39015649089fd200123f040125: No bootstrap required, opened a new log
I20260620 14:48:14.144368 8502 ts_tablet_manager.cc:1397] T 6a36724722064849948c8e1c8b904d24 P 1a858e39015649089fd200123f040125: Time spent bootstrapping tablet: real 0.010s user 0.009s sys 0.000s
I20260620 14:48:14.146965 8504 tablet_bootstrap.cc:492] T 6a36724722064849948c8e1c8b904d24 P 3a2fba6e705d4e23af93e43abe62f350: No bootstrap required, opened a new log
I20260620 14:48:14.147246 8504 ts_tablet_manager.cc:1397] T 6a36724722064849948c8e1c8b904d24 P 3a2fba6e705d4e23af93e43abe62f350: Time spent bootstrapping tablet: real 0.009s user 0.006s sys 0.000s
I20260620 14:48:14.147385 8501 raft_consensus.cc:348] T 6a36724722064849948c8e1c8b904d24 P a99fedfd9f1844aeb7e6362e68c7d072 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a99fedfd9f1844aeb7e6362e68c7d072" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 43441 } } peers { permanent_uuid: "1a858e39015649089fd200123f040125" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 44163 } } peers { permanent_uuid: "3a2fba6e705d4e23af93e43abe62f350" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 39761 } }
I20260620 14:48:14.147882 8501 raft_consensus.cc:374] T 6a36724722064849948c8e1c8b904d24 P a99fedfd9f1844aeb7e6362e68c7d072 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 14:48:14.147989 8501 raft_consensus.cc:729] T 6a36724722064849948c8e1c8b904d24 P a99fedfd9f1844aeb7e6362e68c7d072 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a99fedfd9f1844aeb7e6362e68c7d072, State: Initialized, Role: FOLLOWER
I20260620 14:48:14.148617 8501 consensus_queue.cc:260] T 6a36724722064849948c8e1c8b904d24 P a99fedfd9f1844aeb7e6362e68c7d072 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a99fedfd9f1844aeb7e6362e68c7d072" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 43441 } } peers { permanent_uuid: "1a858e39015649089fd200123f040125" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 44163 } } peers { permanent_uuid: "3a2fba6e705d4e23af93e43abe62f350" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 39761 } }
I20260620 14:48:14.149792 8338 heartbeater.cc:503] Master 127.0.97.62:33203 was elected leader, sending a full tablet report...
I20260620 14:48:14.150848 8501 ts_tablet_manager.cc:1428] T 6a36724722064849948c8e1c8b904d24 P a99fedfd9f1844aeb7e6362e68c7d072: Time spent starting tablet: real 0.012s user 0.012s sys 0.000s
I20260620 14:48:14.155550 8502 raft_consensus.cc:348] T 6a36724722064849948c8e1c8b904d24 P 1a858e39015649089fd200123f040125 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a99fedfd9f1844aeb7e6362e68c7d072" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 43441 } } peers { permanent_uuid: "1a858e39015649089fd200123f040125" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 44163 } } peers { permanent_uuid: "3a2fba6e705d4e23af93e43abe62f350" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 39761 } }
I20260620 14:48:14.156062 8502 raft_consensus.cc:374] T 6a36724722064849948c8e1c8b904d24 P 1a858e39015649089fd200123f040125 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 14:48:14.156168 8502 raft_consensus.cc:729] T 6a36724722064849948c8e1c8b904d24 P 1a858e39015649089fd200123f040125 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1a858e39015649089fd200123f040125, State: Initialized, Role: FOLLOWER
I20260620 14:48:14.156948 8502 consensus_queue.cc:260] T 6a36724722064849948c8e1c8b904d24 P 1a858e39015649089fd200123f040125 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a99fedfd9f1844aeb7e6362e68c7d072" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 43441 } } peers { permanent_uuid: "1a858e39015649089fd200123f040125" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 44163 } } peers { permanent_uuid: "3a2fba6e705d4e23af93e43abe62f350" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 39761 } }
I20260620 14:48:14.157196 8504 raft_consensus.cc:348] T 6a36724722064849948c8e1c8b904d24 P 3a2fba6e705d4e23af93e43abe62f350 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a99fedfd9f1844aeb7e6362e68c7d072" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 43441 } } peers { permanent_uuid: "1a858e39015649089fd200123f040125" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 44163 } } peers { permanent_uuid: "3a2fba6e705d4e23af93e43abe62f350" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 39761 } }
I20260620 14:48:14.157660 8504 raft_consensus.cc:374] T 6a36724722064849948c8e1c8b904d24 P 3a2fba6e705d4e23af93e43abe62f350 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 14:48:14.157763 8504 raft_consensus.cc:729] T 6a36724722064849948c8e1c8b904d24 P 3a2fba6e705d4e23af93e43abe62f350 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3a2fba6e705d4e23af93e43abe62f350, State: Initialized, Role: FOLLOWER
I20260620 14:48:14.158110 8477 heartbeater.cc:503] Master 127.0.97.62:33203 was elected leader, sending a full tablet report...
I20260620 14:48:14.158382 8504 consensus_queue.cc:260] T 6a36724722064849948c8e1c8b904d24 P 3a2fba6e705d4e23af93e43abe62f350 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a99fedfd9f1844aeb7e6362e68c7d072" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 43441 } } peers { permanent_uuid: "1a858e39015649089fd200123f040125" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 44163 } } peers { permanent_uuid: "3a2fba6e705d4e23af93e43abe62f350" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 39761 } }
I20260620 14:48:14.158867 8502 ts_tablet_manager.cc:1428] T 6a36724722064849948c8e1c8b904d24 P 1a858e39015649089fd200123f040125: Time spent starting tablet: real 0.014s user 0.012s sys 0.000s
I20260620 14:48:14.160066 8504 ts_tablet_manager.cc:1428] T 6a36724722064849948c8e1c8b904d24 P 3a2fba6e705d4e23af93e43abe62f350: Time spent starting tablet: real 0.013s user 0.014s sys 0.001s
W20260620 14:48:14.182751 8199 tablet.cc:2369] T 6a36724722064849948c8e1c8b904d24 P 3a2fba6e705d4e23af93e43abe62f350: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260620 14:48:14.234041 8339 tablet.cc:2369] T 6a36724722064849948c8e1c8b904d24 P a99fedfd9f1844aeb7e6362e68c7d072: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260620 14:48:14.243396 8478 tablet.cc:2369] T 6a36724722064849948c8e1c8b904d24 P 1a858e39015649089fd200123f040125: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260620 14:48:14.510207 8507 raft_consensus.cc:482] T 6a36724722064849948c8e1c8b904d24 P a99fedfd9f1844aeb7e6362e68c7d072 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260620 14:48:14.510586 8507 raft_consensus.cc:504] T 6a36724722064849948c8e1c8b904d24 P a99fedfd9f1844aeb7e6362e68c7d072 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a99fedfd9f1844aeb7e6362e68c7d072" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 43441 } } peers { permanent_uuid: "1a858e39015649089fd200123f040125" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 44163 } } peers { permanent_uuid: "3a2fba6e705d4e23af93e43abe62f350" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 39761 } }
I20260620 14:48:14.512642 8507 leader_election.cc:288] T 6a36724722064849948c8e1c8b904d24 P a99fedfd9f1844aeb7e6362e68c7d072 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 1a858e39015649089fd200123f040125 (127.0.97.3:44163), 3a2fba6e705d4e23af93e43abe62f350 (127.0.97.1:39761)
I20260620 14:48:14.519795 8498 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:48:14.512992 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.2:49755 (local address 127.0.97.3:44163)
0620 14:48:14.513282 (+ 290us) server_negotiation.cc:206] Beginning negotiation
0620 14:48:14.513292 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0620 14:48:14.514092 (+ 800us) server_negotiation.cc:407] Connection header received
0620 14:48:14.514372 (+ 280us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:48:14.514376 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:48:14.514432 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:48:14.514537 (+ 105us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 14:48:14.515254 (+ 717us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:48:14.515992 (+ 738us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:48:14.518025 (+ 2033us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:48:14.518767 (+ 742us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:48:14.518886 (+ 119us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:48:14.519021 (+ 135us) server_negotiation.cc:1035] Waiting for connection context
0620 14:48:14.519392 (+ 371us) server_negotiation.cc:299] Negotiation successful
0620 14:48:14.519514 (+ 122us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":95}
I20260620 14:48:14.519809 8499 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:48:14.513306 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.2:48741 (local address 127.0.97.1:39761)
0620 14:48:14.513519 (+ 213us) server_negotiation.cc:206] Beginning negotiation
0620 14:48:14.513528 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0620 14:48:14.514158 (+ 630us) server_negotiation.cc:407] Connection header received
0620 14:48:14.514342 (+ 184us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:48:14.514348 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:48:14.514431 (+ 83us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:48:14.514542 (+ 111us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 14:48:14.515178 (+ 636us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:48:14.516071 (+ 893us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:48:14.517896 (+ 1825us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:48:14.518769 (+ 873us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:48:14.518959 (+ 190us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:48:14.519071 (+ 112us) server_negotiation.cc:1035] Waiting for connection context
0620 14:48:14.519346 (+ 275us) server_negotiation.cc:299] Negotiation successful
0620 14:48:14.519488 (+ 142us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":46}
I20260620 14:48:14.520267 8511 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:48:14.513127 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:39761 (local address 127.0.97.2:48741)
0620 14:48:14.514004 (+ 877us) negotiation.cc:107] Waiting for socket to connect
0620 14:48:14.514019 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0620 14:48:14.514189 (+ 170us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 14:48:14.514556 (+ 367us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 14:48:14.514561 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 14:48:14.514609 (+ 48us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 14:48:14.515003 (+ 394us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:48:14.515011 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:48:14.516214 (+ 1203us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:48:14.516217 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:48:14.517725 (+ 1508us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:48:14.517740 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:48:14.519037 (+ 1297us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:48:14.519046 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:48:14.519177 (+ 131us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:48:14.519200 (+ 23us) client_negotiation.cc:769] Sending connection context
0620 14:48:14.519283 (+ 83us) client_negotiation.cc:240] Negotiation successful
0620 14:48:14.519434 (+ 151us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":716,"mutex_wait_us":24,"spinlock_wait_cycles":46080,"thread_start_us":545,"threads_started":1}
I20260620 14:48:14.520390 8510 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:48:14.512890 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.3:44163 (local address 127.0.97.2:49755)
0620 14:48:14.513933 (+ 1043us) negotiation.cc:107] Waiting for socket to connect
0620 14:48:14.513951 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0620 14:48:14.514189 (+ 238us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 14:48:14.514558 (+ 369us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 14:48:14.514563 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 14:48:14.514614 (+ 51us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 14:48:14.515075 (+ 461us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:48:14.515110 (+ 35us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:48:14.516136 (+ 1026us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:48:14.516145 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:48:14.517863 (+ 1718us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:48:14.517879 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:48:14.519000 (+ 1121us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:48:14.519008 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:48:14.519202 (+ 194us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:48:14.519227 (+ 25us) client_negotiation.cc:769] Sending connection context
0620 14:48:14.519347 (+ 120us) client_negotiation.cc:240] Negotiation successful
0620 14:48:14.519484 (+ 137us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":895,"mutex_wait_us":65,"spinlock_wait_cycles":22912,"thread_start_us":652,"threads_started":1}
I20260620 14:48:14.521420 8427 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6a36724722064849948c8e1c8b904d24" candidate_uuid: "a99fedfd9f1844aeb7e6362e68c7d072" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1a858e39015649089fd200123f040125" is_pre_election: true
I20260620 14:48:14.521435 8153 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6a36724722064849948c8e1c8b904d24" candidate_uuid: "a99fedfd9f1844aeb7e6362e68c7d072" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3a2fba6e705d4e23af93e43abe62f350" is_pre_election: true
I20260620 14:48:14.522002 8427 raft_consensus.cc:2454] T 6a36724722064849948c8e1c8b904d24 P 1a858e39015649089fd200123f040125 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a99fedfd9f1844aeb7e6362e68c7d072 in term 0.
I20260620 14:48:14.522001 8153 raft_consensus.cc:2454] T 6a36724722064849948c8e1c8b904d24 P 3a2fba6e705d4e23af93e43abe62f350 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a99fedfd9f1844aeb7e6362e68c7d072 in term 0.
I20260620 14:48:14.522799 8219 leader_election.cc:302] T 6a36724722064849948c8e1c8b904d24 P a99fedfd9f1844aeb7e6362e68c7d072 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 3 yes votes; 0 no votes. yes voters: 1a858e39015649089fd200123f040125, 3a2fba6e705d4e23af93e43abe62f350, a99fedfd9f1844aeb7e6362e68c7d072; no voters:
I20260620 14:48:14.523337 8507 raft_consensus.cc:2781] T 6a36724722064849948c8e1c8b904d24 P a99fedfd9f1844aeb7e6362e68c7d072 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260620 14:48:14.523515 8507 raft_consensus.cc:482] T 6a36724722064849948c8e1c8b904d24 P a99fedfd9f1844aeb7e6362e68c7d072 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260620 14:48:14.523696 8507 raft_consensus.cc:3037] T 6a36724722064849948c8e1c8b904d24 P a99fedfd9f1844aeb7e6362e68c7d072 [term 0 FOLLOWER]: Advancing to term 1
I20260620 14:48:14.525810 8507 raft_consensus.cc:504] T 6a36724722064849948c8e1c8b904d24 P a99fedfd9f1844aeb7e6362e68c7d072 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a99fedfd9f1844aeb7e6362e68c7d072" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 43441 } } peers { permanent_uuid: "1a858e39015649089fd200123f040125" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 44163 } } peers { permanent_uuid: "3a2fba6e705d4e23af93e43abe62f350" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 39761 } }
I20260620 14:48:14.526443 8507 leader_election.cc:288] T 6a36724722064849948c8e1c8b904d24 P a99fedfd9f1844aeb7e6362e68c7d072 [CANDIDATE]: Term 1 election: Requested vote from peers 1a858e39015649089fd200123f040125 (127.0.97.3:44163), 3a2fba6e705d4e23af93e43abe62f350 (127.0.97.1:39761)
I20260620 14:48:14.526948 8427 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6a36724722064849948c8e1c8b904d24" candidate_uuid: "a99fedfd9f1844aeb7e6362e68c7d072" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1a858e39015649089fd200123f040125"
I20260620 14:48:14.527005 8153 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6a36724722064849948c8e1c8b904d24" candidate_uuid: "a99fedfd9f1844aeb7e6362e68c7d072" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3a2fba6e705d4e23af93e43abe62f350"
I20260620 14:48:14.527194 8427 raft_consensus.cc:3037] T 6a36724722064849948c8e1c8b904d24 P 1a858e39015649089fd200123f040125 [term 0 FOLLOWER]: Advancing to term 1
I20260620 14:48:14.527202 8153 raft_consensus.cc:3037] T 6a36724722064849948c8e1c8b904d24 P 3a2fba6e705d4e23af93e43abe62f350 [term 0 FOLLOWER]: Advancing to term 1
I20260620 14:48:14.529435 8153 raft_consensus.cc:2454] T 6a36724722064849948c8e1c8b904d24 P 3a2fba6e705d4e23af93e43abe62f350 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a99fedfd9f1844aeb7e6362e68c7d072 in term 1.
I20260620 14:48:14.529582 8427 raft_consensus.cc:2454] T 6a36724722064849948c8e1c8b904d24 P 1a858e39015649089fd200123f040125 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a99fedfd9f1844aeb7e6362e68c7d072 in term 1.
I20260620 14:48:14.529960 8219 leader_election.cc:302] T 6a36724722064849948c8e1c8b904d24 P a99fedfd9f1844aeb7e6362e68c7d072 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3a2fba6e705d4e23af93e43abe62f350, a99fedfd9f1844aeb7e6362e68c7d072; no voters:
I20260620 14:48:14.530296 8507 raft_consensus.cc:2781] T 6a36724722064849948c8e1c8b904d24 P a99fedfd9f1844aeb7e6362e68c7d072 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 14:48:14.530709 8507 raft_consensus.cc:686] T 6a36724722064849948c8e1c8b904d24 P a99fedfd9f1844aeb7e6362e68c7d072 [term 1 LEADER]: Becoming Leader. State: Replica: a99fedfd9f1844aeb7e6362e68c7d072, State: Running, Role: LEADER
I20260620 14:48:14.531229 8507 consensus_queue.cc:237] T 6a36724722064849948c8e1c8b904d24 P a99fedfd9f1844aeb7e6362e68c7d072 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a99fedfd9f1844aeb7e6362e68c7d072" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 43441 } } peers { permanent_uuid: "1a858e39015649089fd200123f040125" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 44163 } } peers { permanent_uuid: "3a2fba6e705d4e23af93e43abe62f350" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 39761 } }
I20260620 14:48:14.536232 7989 catalog_manager.cc:5495] T 6a36724722064849948c8e1c8b904d24 P a99fedfd9f1844aeb7e6362e68c7d072 reported cstate change: term changed from 0 to 1, leader changed from <none> to a99fedfd9f1844aeb7e6362e68c7d072 (127.0.97.2). New cstate: current_term: 1 leader_uuid: "a99fedfd9f1844aeb7e6362e68c7d072" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a99fedfd9f1844aeb7e6362e68c7d072" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 43441 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "1a858e39015649089fd200123f040125" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 44163 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "3a2fba6e705d4e23af93e43abe62f350" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 39761 } health_report { overall_health: UNKNOWN } } }
I20260620 14:48:14.558303 8496 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:48:14.553555 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36916 (local address 127.0.97.2:43441)
0620 14:48:14.553724 (+ 169us) server_negotiation.cc:206] Beginning negotiation
0620 14:48:14.553729 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 14:48:14.553746 (+ 17us) server_negotiation.cc:407] Connection header received
0620 14:48:14.553942 (+ 196us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:48:14.553946 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:48:14.554021 (+ 75us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:48:14.554145 (+ 124us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 14:48:14.554439 (+ 294us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:48:14.555154 (+ 715us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:48:14.556504 (+ 1350us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:48:14.556878 (+ 374us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:48:14.556972 (+ 94us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:48:14.557329 (+ 357us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 14:48:14.557860 (+ 531us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0620 14:48:14.557964 (+ 104us) server_negotiation.cc:1035] Waiting for connection context
0620 14:48:14.558034 (+ 70us) server_negotiation.cc:299] Negotiation successful
0620 14:48:14.558095 (+ 61us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":52}
I20260620 14:48:14.566414 8427 raft_consensus.cc:1261] T 6a36724722064849948c8e1c8b904d24 P 1a858e39015649089fd200123f040125 [term 1 FOLLOWER]: Refusing update from remote peer a99fedfd9f1844aeb7e6362e68c7d072: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 14:48:14.566432 8153 raft_consensus.cc:1261] T 6a36724722064849948c8e1c8b904d24 P 3a2fba6e705d4e23af93e43abe62f350 [term 1 FOLLOWER]: Refusing update from remote peer a99fedfd9f1844aeb7e6362e68c7d072: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 14:48:14.567376 8512 consensus_queue.cc:1035] T 6a36724722064849948c8e1c8b904d24 P a99fedfd9f1844aeb7e6362e68c7d072 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1a858e39015649089fd200123f040125" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 44163 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 14:48:14.567693 8507 consensus_queue.cc:1035] T 6a36724722064849948c8e1c8b904d24 P a99fedfd9f1844aeb7e6362e68c7d072 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3a2fba6e705d4e23af93e43abe62f350" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 39761 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 14:48:14.577256 8515 mvcc.cc:205] Tried to move back new op lower bound from 7298936400134287360 to 7298936400004210688. Current Snapshot: MvccSnapshot[applied={T|T < 7298936400134287360}]
I20260620 14:48:14.578040 8516 mvcc.cc:205] Tried to move back new op lower bound from 7298936400134287360 to 7298936400004210688. Current Snapshot: MvccSnapshot[applied={T|T < 7298936400134287360}]
I20260620 14:48:14.593418 7989 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:52204:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 14:48:14.593647 7989 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:52204:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 14:48:14.597175 7989 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 0af6e58150ce4b73a4c412dd6c640a60: Sending DeleteTablet for 3 replicas of tablet 6a36724722064849948c8e1c8b904d24
I20260620 14:48:14.598351 8268 tablet_service.cc:1514] Processing DeleteTablet for tablet 6a36724722064849948c8e1c8b904d24 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 14:48:14 UTC) from {username='oryx', principal='oryx/127.0.97.62@KRBTEST.COM'} at 127.0.0.1:36912
I20260620 14:48:14.598338 8407 tablet_service.cc:1514] Processing DeleteTablet for tablet 6a36724722064849948c8e1c8b904d24 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 14:48:14 UTC) from {username='oryx', principal='oryx/127.0.97.62@KRBTEST.COM'} at 127.0.0.1:33502
I20260620 14:48:14.598558 8128 tablet_service.cc:1514] Processing DeleteTablet for tablet 6a36724722064849948c8e1c8b904d24 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 14:48:14 UTC) from {username='oryx', principal='oryx/127.0.97.62@KRBTEST.COM'} at 127.0.0.1:56560
I20260620 14:48:14.600932 7989 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:52204:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260620 14:48:14.601879 7989 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260620 14:48:14.609213 8527 tablet_replica.cc:330] T 6a36724722064849948c8e1c8b904d24 P 3a2fba6e705d4e23af93e43abe62f350: stopping tablet replica
I20260620 14:48:14.609753 8527 raft_consensus.cc:2229] T 6a36724722064849948c8e1c8b904d24 P 3a2fba6e705d4e23af93e43abe62f350 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 14:48:14.610332 8527 raft_consensus.cc:2258] T 6a36724722064849948c8e1c8b904d24 P 3a2fba6e705d4e23af93e43abe62f350 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 14:48:14.614672 8527 ts_tablet_manager.cc:1905] T 6a36724722064849948c8e1c8b904d24 P 3a2fba6e705d4e23af93e43abe62f350: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 14:48:14.615706 8128 tablet_service.cc:1467] Processing CreateTablet for tablet f1bb588ed8704523b41ee390ac5f5506 (DEFAULT_TABLE table=test-table [id=7186025b0521402484b97a14988e2514]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 14:48:14.615772 8267 tablet_service.cc:1467] Processing CreateTablet for tablet f1bb588ed8704523b41ee390ac5f5506 (DEFAULT_TABLE table=test-table [id=7186025b0521402484b97a14988e2514]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 14:48:14.616160 8128 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f1bb588ed8704523b41ee390ac5f5506. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 14:48:14.616868 8267 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f1bb588ed8704523b41ee390ac5f5506. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 14:48:14.619575 8525 tablet_replica.cc:330] T 6a36724722064849948c8e1c8b904d24 P 1a858e39015649089fd200123f040125: stopping tablet replica
I20260620 14:48:14.620184 8525 raft_consensus.cc:2229] T 6a36724722064849948c8e1c8b904d24 P 1a858e39015649089fd200123f040125 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 14:48:14.620718 8525 raft_consensus.cc:2258] T 6a36724722064849948c8e1c8b904d24 P 1a858e39015649089fd200123f040125 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 14:48:14.623824 8406 tablet_service.cc:1467] Processing CreateTablet for tablet f1bb588ed8704523b41ee390ac5f5506 (DEFAULT_TABLE table=test-table [id=7186025b0521402484b97a14988e2514]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 14:48:14.624271 8406 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f1bb588ed8704523b41ee390ac5f5506. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 14:48:14.624430 8525 ts_tablet_manager.cc:1905] T 6a36724722064849948c8e1c8b904d24 P 1a858e39015649089fd200123f040125: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 14:48:14.625528 8526 tablet_replica.cc:330] T 6a36724722064849948c8e1c8b904d24 P a99fedfd9f1844aeb7e6362e68c7d072: stopping tablet replica
I20260620 14:48:14.625957 8526 raft_consensus.cc:2229] T 6a36724722064849948c8e1c8b904d24 P a99fedfd9f1844aeb7e6362e68c7d072 [term 1 LEADER]: Raft consensus shutting down.
I20260620 14:48:14.626627 8526 raft_consensus.cc:2258] T 6a36724722064849948c8e1c8b904d24 P a99fedfd9f1844aeb7e6362e68c7d072 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 14:48:14.628831 8526 ts_tablet_manager.cc:1905] T 6a36724722064849948c8e1c8b904d24 P a99fedfd9f1844aeb7e6362e68c7d072: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 14:48:14.629976 8501 tablet_bootstrap.cc:492] T f1bb588ed8704523b41ee390ac5f5506 P a99fedfd9f1844aeb7e6362e68c7d072: Bootstrap starting.
I20260620 14:48:14.630553 8504 tablet_bootstrap.cc:492] T f1bb588ed8704523b41ee390ac5f5506 P 3a2fba6e705d4e23af93e43abe62f350: Bootstrap starting.
I20260620 14:48:14.630658 8502 tablet_bootstrap.cc:492] T f1bb588ed8704523b41ee390ac5f5506 P 1a858e39015649089fd200123f040125: Bootstrap starting.
I20260620 14:48:14.631757 8501 tablet_bootstrap.cc:654] T f1bb588ed8704523b41ee390ac5f5506 P a99fedfd9f1844aeb7e6362e68c7d072: Neither blocks nor log segments found. Creating new log.
I20260620 14:48:14.632218 8502 tablet_bootstrap.cc:654] T f1bb588ed8704523b41ee390ac5f5506 P 1a858e39015649089fd200123f040125: Neither blocks nor log segments found. Creating new log.
I20260620 14:48:14.632218 8504 tablet_bootstrap.cc:654] T f1bb588ed8704523b41ee390ac5f5506 P 3a2fba6e705d4e23af93e43abe62f350: Neither blocks nor log segments found. Creating new log.
I20260620 14:48:14.637795 8526 ts_tablet_manager.cc:1918] T 6a36724722064849948c8e1c8b904d24 P a99fedfd9f1844aeb7e6362e68c7d072: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260620 14:48:14.638033 8526 log.cc:1196] T 6a36724722064849948c8e1c8b904d24 P a99fedfd9f1844aeb7e6362e68c7d072: Deleting WAL directory at /tmp/dist-test-taskUci3DU/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781966829091300-388-0/minicluster-data/ts-1/wal/wals/6a36724722064849948c8e1c8b904d24
I20260620 14:48:14.639119 8504 tablet_bootstrap.cc:492] T f1bb588ed8704523b41ee390ac5f5506 P 3a2fba6e705d4e23af93e43abe62f350: No bootstrap required, opened a new log
I20260620 14:48:14.639392 8504 ts_tablet_manager.cc:1397] T f1bb588ed8704523b41ee390ac5f5506 P 3a2fba6e705d4e23af93e43abe62f350: Time spent bootstrapping tablet: real 0.009s user 0.000s sys 0.005s
I20260620 14:48:14.639139 8527 ts_tablet_manager.cc:1918] T 6a36724722064849948c8e1c8b904d24 P 3a2fba6e705d4e23af93e43abe62f350: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260620 14:48:14.639853 8527 log.cc:1196] T 6a36724722064849948c8e1c8b904d24 P 3a2fba6e705d4e23af93e43abe62f350: Deleting WAL directory at /tmp/dist-test-taskUci3DU/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781966829091300-388-0/minicluster-data/ts-0/wal/wals/6a36724722064849948c8e1c8b904d24
I20260620 14:48:14.640432 8504 raft_consensus.cc:348] T f1bb588ed8704523b41ee390ac5f5506 P 3a2fba6e705d4e23af93e43abe62f350 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3a2fba6e705d4e23af93e43abe62f350" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 39761 } } peers { permanent_uuid: "a99fedfd9f1844aeb7e6362e68c7d072" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 43441 } } peers { permanent_uuid: "1a858e39015649089fd200123f040125" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 44163 } }
I20260620 14:48:14.640648 8504 raft_consensus.cc:374] T f1bb588ed8704523b41ee390ac5f5506 P 3a2fba6e705d4e23af93e43abe62f350 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 14:48:14.640745 8504 raft_consensus.cc:729] T f1bb588ed8704523b41ee390ac5f5506 P 3a2fba6e705d4e23af93e43abe62f350 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3a2fba6e705d4e23af93e43abe62f350, State: Initialized, Role: FOLLOWER
I20260620 14:48:14.641085 8504 consensus_queue.cc:260] T f1bb588ed8704523b41ee390ac5f5506 P 3a2fba6e705d4e23af93e43abe62f350 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3a2fba6e705d4e23af93e43abe62f350" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 39761 } } peers { permanent_uuid: "a99fedfd9f1844aeb7e6362e68c7d072" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 43441 } } peers { permanent_uuid: "1a858e39015649089fd200123f040125" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 44163 } }
I20260620 14:48:14.642477 8504 ts_tablet_manager.cc:1428] T f1bb588ed8704523b41ee390ac5f5506 P 3a2fba6e705d4e23af93e43abe62f350: Time spent starting tablet: real 0.003s user 0.002s sys 0.000s
I20260620 14:48:14.643148 8526 ts_tablet_manager.cc:1939] T 6a36724722064849948c8e1c8b904d24 P a99fedfd9f1844aeb7e6362e68c7d072: Deleting consensus metadata
I20260620 14:48:14.643740 8527 ts_tablet_manager.cc:1939] T 6a36724722064849948c8e1c8b904d24 P 3a2fba6e705d4e23af93e43abe62f350: Deleting consensus metadata
I20260620 14:48:14.645067 7977 catalog_manager.cc:4841] TS a99fedfd9f1844aeb7e6362e68c7d072 (127.0.97.2:43441): tablet 6a36724722064849948c8e1c8b904d24 (table test-table [id=ee5ba9f2145b42099311eb4507b636d4]) successfully deleted
I20260620 14:48:14.645457 7976 catalog_manager.cc:4841] TS 3a2fba6e705d4e23af93e43abe62f350 (127.0.97.1:39761): tablet 6a36724722064849948c8e1c8b904d24 (table test-table [id=ee5ba9f2145b42099311eb4507b636d4]) successfully deleted
I20260620 14:48:14.646622 8525 ts_tablet_manager.cc:1918] T 6a36724722064849948c8e1c8b904d24 P 1a858e39015649089fd200123f040125: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260620 14:48:14.646829 8525 log.cc:1196] T 6a36724722064849948c8e1c8b904d24 P 1a858e39015649089fd200123f040125: Deleting WAL directory at /tmp/dist-test-taskUci3DU/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781966829091300-388-0/minicluster-data/ts-2/wal/wals/6a36724722064849948c8e1c8b904d24
I20260620 14:48:14.647123 8501 tablet_bootstrap.cc:492] T f1bb588ed8704523b41ee390ac5f5506 P a99fedfd9f1844aeb7e6362e68c7d072: No bootstrap required, opened a new log
I20260620 14:48:14.647295 8501 ts_tablet_manager.cc:1397] T f1bb588ed8704523b41ee390ac5f5506 P a99fedfd9f1844aeb7e6362e68c7d072: Time spent bootstrapping tablet: real 0.017s user 0.001s sys 0.007s
I20260620 14:48:14.647387 8525 ts_tablet_manager.cc:1939] T 6a36724722064849948c8e1c8b904d24 P 1a858e39015649089fd200123f040125: Deleting consensus metadata
I20260620 14:48:14.647536 8502 tablet_bootstrap.cc:492] T f1bb588ed8704523b41ee390ac5f5506 P 1a858e39015649089fd200123f040125: No bootstrap required, opened a new log
I20260620 14:48:14.647716 8502 ts_tablet_manager.cc:1397] T f1bb588ed8704523b41ee390ac5f5506 P 1a858e39015649089fd200123f040125: Time spent bootstrapping tablet: real 0.017s user 0.002s sys 0.009s
I20260620 14:48:14.648191 8501 raft_consensus.cc:348] T f1bb588ed8704523b41ee390ac5f5506 P a99fedfd9f1844aeb7e6362e68c7d072 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3a2fba6e705d4e23af93e43abe62f350" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 39761 } } peers { permanent_uuid: "a99fedfd9f1844aeb7e6362e68c7d072" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 43441 } } peers { permanent_uuid: "1a858e39015649089fd200123f040125" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 44163 } }
I20260620 14:48:14.648440 8501 raft_consensus.cc:374] T f1bb588ed8704523b41ee390ac5f5506 P a99fedfd9f1844aeb7e6362e68c7d072 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 14:48:14.648541 8501 raft_consensus.cc:729] T f1bb588ed8704523b41ee390ac5f5506 P a99fedfd9f1844aeb7e6362e68c7d072 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a99fedfd9f1844aeb7e6362e68c7d072, State: Initialized, Role: FOLLOWER
I20260620 14:48:14.648562 8502 raft_consensus.cc:348] T f1bb588ed8704523b41ee390ac5f5506 P 1a858e39015649089fd200123f040125 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3a2fba6e705d4e23af93e43abe62f350" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 39761 } } peers { permanent_uuid: "a99fedfd9f1844aeb7e6362e68c7d072" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 43441 } } peers { permanent_uuid: "1a858e39015649089fd200123f040125" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 44163 } }
I20260620 14:48:14.648788 8502 raft_consensus.cc:374] T f1bb588ed8704523b41ee390ac5f5506 P 1a858e39015649089fd200123f040125 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 14:48:14.648886 8502 raft_consensus.cc:729] T f1bb588ed8704523b41ee390ac5f5506 P 1a858e39015649089fd200123f040125 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1a858e39015649089fd200123f040125, State: Initialized, Role: FOLLOWER
I20260620 14:48:14.648851 8501 consensus_queue.cc:260] T f1bb588ed8704523b41ee390ac5f5506 P a99fedfd9f1844aeb7e6362e68c7d072 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3a2fba6e705d4e23af93e43abe62f350" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 39761 } } peers { permanent_uuid: "a99fedfd9f1844aeb7e6362e68c7d072" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 43441 } } peers { permanent_uuid: "1a858e39015649089fd200123f040125" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 44163 } }
I20260620 14:48:14.649179 8502 consensus_queue.cc:260] T f1bb588ed8704523b41ee390ac5f5506 P 1a858e39015649089fd200123f040125 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3a2fba6e705d4e23af93e43abe62f350" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 39761 } } peers { permanent_uuid: "a99fedfd9f1844aeb7e6362e68c7d072" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 43441 } } peers { permanent_uuid: "1a858e39015649089fd200123f040125" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 44163 } }
I20260620 14:48:14.649760 8502 ts_tablet_manager.cc:1428] T f1bb588ed8704523b41ee390ac5f5506 P 1a858e39015649089fd200123f040125: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20260620 14:48:14.651084 7975 catalog_manager.cc:4841] TS 1a858e39015649089fd200123f040125 (127.0.97.3:44163): tablet 6a36724722064849948c8e1c8b904d24 (table test-table [id=ee5ba9f2145b42099311eb4507b636d4]) successfully deleted
I20260620 14:48:14.651437 8501 ts_tablet_manager.cc:1428] T f1bb588ed8704523b41ee390ac5f5506 P a99fedfd9f1844aeb7e6362e68c7d072: Time spent starting tablet: real 0.004s user 0.003s sys 0.000s
I20260620 14:48:14.973443 8508 raft_consensus.cc:482] T f1bb588ed8704523b41ee390ac5f5506 P 1a858e39015649089fd200123f040125 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260620 14:48:14.973694 8508 raft_consensus.cc:504] T f1bb588ed8704523b41ee390ac5f5506 P 1a858e39015649089fd200123f040125 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3a2fba6e705d4e23af93e43abe62f350" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 39761 } } peers { permanent_uuid: "a99fedfd9f1844aeb7e6362e68c7d072" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 43441 } } peers { permanent_uuid: "1a858e39015649089fd200123f040125" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 44163 } }
I20260620 14:48:14.975354 8508 leader_election.cc:288] T f1bb588ed8704523b41ee390ac5f5506 P 1a858e39015649089fd200123f040125 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 3a2fba6e705d4e23af93e43abe62f350 (127.0.97.1:39761), a99fedfd9f1844aeb7e6362e68c7d072 (127.0.97.2:43441)
I20260620 14:48:14.984989 8499 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:48:14.976458 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.3:40165 (local address 127.0.97.1:39761)
0620 14:48:14.976651 (+ 193us) server_negotiation.cc:206] Beginning negotiation
0620 14:48:14.976658 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0620 14:48:14.977146 (+ 488us) server_negotiation.cc:407] Connection header received
0620 14:48:14.977459 (+ 313us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:48:14.977465 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:48:14.977590 (+ 125us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:48:14.977701 (+ 111us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 14:48:14.978877 (+ 1176us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:48:14.980202 (+ 1325us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:48:14.982728 (+ 2526us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:48:14.983797 (+ 1069us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:48:14.983938 (+ 141us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:48:14.984049 (+ 111us) server_negotiation.cc:1035] Waiting for connection context
0620 14:48:14.984410 (+ 361us) server_negotiation.cc:299] Negotiation successful
0620 14:48:14.984554 (+ 144us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":51}
I20260620 14:48:14.985034 8531 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:48:14.975573 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:39761 (local address 127.0.97.3:40165)
0620 14:48:14.977008 (+ 1435us) negotiation.cc:107] Waiting for socket to connect
0620 14:48:14.977033 (+ 25us) client_negotiation.cc:174] Beginning negotiation
0620 14:48:14.977279 (+ 246us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 14:48:14.977738 (+ 459us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 14:48:14.977742 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 14:48:14.977765 (+ 23us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 14:48:14.978158 (+ 393us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:48:14.978168 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:48:14.980350 (+ 2182us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:48:14.980355 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:48:14.982492 (+ 2137us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:48:14.982501 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:48:14.983996 (+ 1495us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:48:14.984002 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:48:14.984224 (+ 222us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:48:14.984266 (+ 42us) client_negotiation.cc:769] Sending connection context
0620 14:48:14.984367 (+ 101us) client_negotiation.cc:240] Negotiation successful
0620 14:48:14.984521 (+ 154us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1284,"mutex_wait_us":33,"thread_start_us":1101,"threads_started":1}
I20260620 14:48:14.985083 8496 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:48:14.975880 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.3:49203 (local address 127.0.97.2:43441)
0620 14:48:14.976117 (+ 237us) server_negotiation.cc:206] Beginning negotiation
0620 14:48:14.976126 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0620 14:48:14.977517 (+ 1391us) server_negotiation.cc:407] Connection header received
0620 14:48:14.977742 (+ 225us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:48:14.977747 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:48:14.977842 (+ 95us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:48:14.977944 (+ 102us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 14:48:14.978706 (+ 762us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:48:14.979919 (+ 1213us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:48:14.982663 (+ 2744us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:48:14.984025 (+ 1362us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:48:14.984166 (+ 141us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:48:14.984270 (+ 104us) server_negotiation.cc:1035] Waiting for connection context
0620 14:48:14.984581 (+ 311us) server_negotiation.cc:299] Negotiation successful
0620 14:48:14.984722 (+ 141us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":76}
I20260620 14:48:14.985924 8532 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:48:14.975814 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.2:43441 (local address 127.0.97.3:49203)
0620 14:48:14.977398 (+ 1584us) negotiation.cc:107] Waiting for socket to connect
0620 14:48:14.977416 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0620 14:48:14.977593 (+ 177us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 14:48:14.977989 (+ 396us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 14:48:14.977996 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 14:48:14.978031 (+ 35us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 14:48:14.978492 (+ 461us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:48:14.978504 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:48:14.980057 (+ 1553us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:48:14.980063 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:48:14.982483 (+ 2420us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:48:14.982502 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:48:14.984234 (+ 1732us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:48:14.984239 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:48:14.984411 (+ 172us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:48:14.984439 (+ 28us) client_negotiation.cc:769] Sending connection context
0620 14:48:14.985453 (+ 1014us) client_negotiation.cc:240] Negotiation successful
0620 14:48:14.985590 (+ 137us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1458,"mutex_wait_us":75,"thread_start_us":1188,"threads_started":1}
I20260620 14:48:14.986090 8153 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f1bb588ed8704523b41ee390ac5f5506" candidate_uuid: "1a858e39015649089fd200123f040125" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3a2fba6e705d4e23af93e43abe62f350" is_pre_election: true
I20260620 14:48:14.986384 8153 raft_consensus.cc:2454] T f1bb588ed8704523b41ee390ac5f5506 P 3a2fba6e705d4e23af93e43abe62f350 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1a858e39015649089fd200123f040125 in term 0.
I20260620 14:48:14.986881 8359 leader_election.cc:302] T f1bb588ed8704523b41ee390ac5f5506 P 1a858e39015649089fd200123f040125 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1a858e39015649089fd200123f040125, 3a2fba6e705d4e23af93e43abe62f350; no voters:
I20260620 14:48:14.986960 8288 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f1bb588ed8704523b41ee390ac5f5506" candidate_uuid: "1a858e39015649089fd200123f040125" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a99fedfd9f1844aeb7e6362e68c7d072" is_pre_election: true
I20260620 14:48:14.987318 8508 raft_consensus.cc:2781] T f1bb588ed8704523b41ee390ac5f5506 P 1a858e39015649089fd200123f040125 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260620 14:48:14.987409 8508 raft_consensus.cc:482] T f1bb588ed8704523b41ee390ac5f5506 P 1a858e39015649089fd200123f040125 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260620 14:48:14.987465 8508 raft_consensus.cc:3037] T f1bb588ed8704523b41ee390ac5f5506 P 1a858e39015649089fd200123f040125 [term 0 FOLLOWER]: Advancing to term 1
I20260620 14:48:14.987408 8288 raft_consensus.cc:2454] T f1bb588ed8704523b41ee390ac5f5506 P a99fedfd9f1844aeb7e6362e68c7d072 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1a858e39015649089fd200123f040125 in term 0.
I20260620 14:48:14.989876 8508 raft_consensus.cc:504] T f1bb588ed8704523b41ee390ac5f5506 P 1a858e39015649089fd200123f040125 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3a2fba6e705d4e23af93e43abe62f350" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 39761 } } peers { permanent_uuid: "a99fedfd9f1844aeb7e6362e68c7d072" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 43441 } } peers { permanent_uuid: "1a858e39015649089fd200123f040125" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 44163 } }
I20260620 14:48:14.990557 8508 leader_election.cc:288] T f1bb588ed8704523b41ee390ac5f5506 P 1a858e39015649089fd200123f040125 [CANDIDATE]: Term 1 election: Requested vote from peers 3a2fba6e705d4e23af93e43abe62f350 (127.0.97.1:39761), a99fedfd9f1844aeb7e6362e68c7d072 (127.0.97.2:43441)
I20260620 14:48:14.990989 8153 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f1bb588ed8704523b41ee390ac5f5506" candidate_uuid: "1a858e39015649089fd200123f040125" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3a2fba6e705d4e23af93e43abe62f350"
I20260620 14:48:14.991206 8153 raft_consensus.cc:3037] T f1bb588ed8704523b41ee390ac5f5506 P 3a2fba6e705d4e23af93e43abe62f350 [term 0 FOLLOWER]: Advancing to term 1
I20260620 14:48:14.993315 8153 raft_consensus.cc:2454] T f1bb588ed8704523b41ee390ac5f5506 P 3a2fba6e705d4e23af93e43abe62f350 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1a858e39015649089fd200123f040125 in term 1.
I20260620 14:48:14.993799 8359 leader_election.cc:302] T f1bb588ed8704523b41ee390ac5f5506 P 1a858e39015649089fd200123f040125 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1a858e39015649089fd200123f040125, 3a2fba6e705d4e23af93e43abe62f350; no voters:
I20260620 14:48:14.994124 8508 raft_consensus.cc:2781] T f1bb588ed8704523b41ee390ac5f5506 P 1a858e39015649089fd200123f040125 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 14:48:14.994675 8508 raft_consensus.cc:686] T f1bb588ed8704523b41ee390ac5f5506 P 1a858e39015649089fd200123f040125 [term 1 LEADER]: Becoming Leader. State: Replica: 1a858e39015649089fd200123f040125, State: Running, Role: LEADER
I20260620 14:48:14.995834 8508 consensus_queue.cc:237] T f1bb588ed8704523b41ee390ac5f5506 P 1a858e39015649089fd200123f040125 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3a2fba6e705d4e23af93e43abe62f350" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 39761 } } peers { permanent_uuid: "a99fedfd9f1844aeb7e6362e68c7d072" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 43441 } } peers { permanent_uuid: "1a858e39015649089fd200123f040125" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 44163 } }
I20260620 14:48:14.999387 8288 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f1bb588ed8704523b41ee390ac5f5506" candidate_uuid: "1a858e39015649089fd200123f040125" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a99fedfd9f1844aeb7e6362e68c7d072"
I20260620 14:48:14.999635 8288 raft_consensus.cc:3037] T f1bb588ed8704523b41ee390ac5f5506 P a99fedfd9f1844aeb7e6362e68c7d072 [term 0 FOLLOWER]: Advancing to term 1
I20260620 14:48:14.999663 7989 catalog_manager.cc:5495] T f1bb588ed8704523b41ee390ac5f5506 P 1a858e39015649089fd200123f040125 reported cstate change: term changed from 0 to 1, leader changed from <none> to 1a858e39015649089fd200123f040125 (127.0.97.3). New cstate: current_term: 1 leader_uuid: "1a858e39015649089fd200123f040125" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3a2fba6e705d4e23af93e43abe62f350" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 39761 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "a99fedfd9f1844aeb7e6362e68c7d072" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 43441 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "1a858e39015649089fd200123f040125" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 44163 } health_report { overall_health: HEALTHY } } }
I20260620 14:48:15.002864 8288 raft_consensus.cc:2454] T f1bb588ed8704523b41ee390ac5f5506 P a99fedfd9f1844aeb7e6362e68c7d072 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1a858e39015649089fd200123f040125 in term 1.
Jun 20 14:48:15 dist-test-slave-q9j6 krb5kdc[7946](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781966892, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.97.62@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM
I20260620 14:48:15.132978 8540 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:48:15.114572 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:33203 (local address 127.0.0.1:52216)
0620 14:48:15.115777 (+ 1205us) negotiation.cc:107] Waiting for socket to connect
0620 14:48:15.115797 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0620 14:48:15.115969 (+ 172us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 14:48:15.116398 (+ 429us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 14:48:15.116409 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 14:48:15.117143 (+ 734us) client_negotiation.cc:189] Negotiated authn=SASL
0620 14:48:15.117393 (+ 250us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:48:15.117402 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:48:15.118496 (+ 1094us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:48:15.118502 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:48:15.119485 (+ 983us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:48:15.119497 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:48:15.120265 (+ 768us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:48:15.120270 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:48:15.120444 (+ 174us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:48:15.124687 (+ 4243us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 14:48:15.124708 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0620 14:48:15.127044 (+ 2336us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 14:48:15.130106 (+ 3062us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 14:48:15.130121 (+ 15us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 14:48:15.130130 (+ 9us) client_negotiation.cc:760] calling sasl_client_step()
0620 14:48:15.130567 (+ 437us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 14:48:15.130956 (+ 389us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 14:48:15.130961 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 14:48:15.130964 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0620 14:48:15.131107 (+ 143us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 14:48:15.131670 (+ 563us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 14:48:15.131679 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 14:48:15.131857 (+ 178us) client_negotiation.cc:769] Sending connection context
0620 14:48:15.132072 (+ 215us) client_negotiation.cc:240] Negotiation successful
0620 14:48:15.132431 (+ 359us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":983,"mutex_wait_us":1,"thread_start_us":790,"threads_started":1}
I20260620 14:48:15.133160 8541 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:48:15.115009 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52216 (local address 127.0.97.62:33203)
0620 14:48:15.115770 (+ 761us) server_negotiation.cc:206] Beginning negotiation
0620 14:48:15.115774 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 14:48:15.115884 (+ 110us) server_negotiation.cc:407] Connection header received
0620 14:48:15.116119 (+ 235us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:48:15.116122 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:48:15.116234 (+ 112us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:48:15.116357 (+ 123us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:48:15.117604 (+ 1247us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:48:15.118355 (+ 751us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:48:15.119661 (+ 1306us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:48:15.120074 (+ 413us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:48:15.120219 (+ 145us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:48:15.127318 (+ 7099us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 14:48:15.127323 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 14:48:15.127326 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 14:48:15.127360 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0620 14:48:15.129941 (+ 2581us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:48:15.130699 (+ 758us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:48:15.130706 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:48:15.130710 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:48:15.130835 (+ 125us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:48:15.131213 (+ 378us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:48:15.131221 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:48:15.131225 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:48:15.131499 (+ 274us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 14:48:15.131674 (+ 175us) server_negotiation.cc:1035] Waiting for connection context
0620 14:48:15.132156 (+ 482us) server_negotiation.cc:299] Negotiation successful
0620 14:48:15.132608 (+ 452us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":193,"server-negotiator.queue_time_us":635,"thread_start_us":377,"threads_started":1}
I20260620 14:48:15.139561 7989 catalog_manager.cc:1908] Servicing CreateTable request from {username='oryx', principal='oryx/127.0.97.62@KRBTEST.COM'} at 127.0.0.1:52216:
name: "kudu_system.kudu_transactions"
schema {
columns {
name: "txn_id"
type: INT64
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "entry_type"
type: INT8
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "identifier"
type: STRING
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "metadata"
type: STRING
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
rows: "<redacted>""\006\001\000\000\000\000\000\000\000\000\007\001@B\017\000\000\000\000\000"
indirect_data: "<redacted>"""
}
partition_schema {
range_schema {
columns {
name: "txn_id"
}
}
}
table_type: TXN_STATUS_TABLE
W20260620 14:48:15.140910 7989 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table kudu_system.kudu_transactions in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260620 14:48:15.148373 8406 tablet_service.cc:1467] Processing CreateTablet for tablet 8b684222965c468b840a0776167ddb41 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=073447d4e3a64e09b635d9c77fa1e3fc]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260620 14:48:15.148303 8128 tablet_service.cc:1467] Processing CreateTablet for tablet 8b684222965c468b840a0776167ddb41 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=073447d4e3a64e09b635d9c77fa1e3fc]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260620 14:48:15.148880 8406 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8b684222965c468b840a0776167ddb41. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 14:48:15.148895 8128 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8b684222965c468b840a0776167ddb41. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 14:48:15.151429 8268 tablet_service.cc:1467] Processing CreateTablet for tablet 8b684222965c468b840a0776167ddb41 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=073447d4e3a64e09b635d9c77fa1e3fc]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260620 14:48:15.152024 8268 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8b684222965c468b840a0776167ddb41. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 14:48:15.156417 8542 tablet_bootstrap.cc:492] T 8b684222965c468b840a0776167ddb41 P 1a858e39015649089fd200123f040125: Bootstrap starting.
I20260620 14:48:15.157825 8542 tablet_bootstrap.cc:654] T 8b684222965c468b840a0776167ddb41 P 1a858e39015649089fd200123f040125: Neither blocks nor log segments found. Creating new log.
I20260620 14:48:15.159847 8542 tablet_bootstrap.cc:492] T 8b684222965c468b840a0776167ddb41 P 1a858e39015649089fd200123f040125: No bootstrap required, opened a new log
I20260620 14:48:15.160006 8542 ts_tablet_manager.cc:1397] T 8b684222965c468b840a0776167ddb41 P 1a858e39015649089fd200123f040125: Time spent bootstrapping tablet: real 0.004s user 0.003s sys 0.000s
I20260620 14:48:15.160642 8544 tablet_bootstrap.cc:492] T 8b684222965c468b840a0776167ddb41 P 3a2fba6e705d4e23af93e43abe62f350: Bootstrap starting.
I20260620 14:48:15.160632 8542 raft_consensus.cc:348] T 8b684222965c468b840a0776167ddb41 P 1a858e39015649089fd200123f040125 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3a2fba6e705d4e23af93e43abe62f350" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 39761 } } peers { permanent_uuid: "a99fedfd9f1844aeb7e6362e68c7d072" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 43441 } } peers { permanent_uuid: "1a858e39015649089fd200123f040125" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 44163 } }
I20260620 14:48:15.160897 8542 raft_consensus.cc:374] T 8b684222965c468b840a0776167ddb41 P 1a858e39015649089fd200123f040125 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 14:48:15.161019 8542 raft_consensus.cc:729] T 8b684222965c468b840a0776167ddb41 P 1a858e39015649089fd200123f040125 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1a858e39015649089fd200123f040125, State: Initialized, Role: FOLLOWER
I20260620 14:48:15.161276 8542 consensus_queue.cc:260] T 8b684222965c468b840a0776167ddb41 P 1a858e39015649089fd200123f040125 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3a2fba6e705d4e23af93e43abe62f350" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 39761 } } peers { permanent_uuid: "a99fedfd9f1844aeb7e6362e68c7d072" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 43441 } } peers { permanent_uuid: "1a858e39015649089fd200123f040125" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 44163 } }
I20260620 14:48:15.161818 8542 ts_tablet_manager.cc:1428] T 8b684222965c468b840a0776167ddb41 P 1a858e39015649089fd200123f040125: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20260620 14:48:15.161676 8508 tablet_replica.cc:439] T 8b684222965c468b840a0776167ddb41 P 1a858e39015649089fd200123f040125: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3a2fba6e705d4e23af93e43abe62f350" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 39761 } } peers { permanent_uuid: "a99fedfd9f1844aeb7e6362e68c7d072" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 43441 } } peers { permanent_uuid: "1a858e39015649089fd200123f040125" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 44163 } } }
I20260620 14:48:15.161974 8508 tablet_replica.cc:442] T 8b684222965c468b840a0776167ddb41 P 1a858e39015649089fd200123f040125: This TxnStatusTablet replica's current role is: FOLLOWER
I20260620 14:48:15.162580 8544 tablet_bootstrap.cc:654] T 8b684222965c468b840a0776167ddb41 P 3a2fba6e705d4e23af93e43abe62f350: Neither blocks nor log segments found. Creating new log.
I20260620 14:48:15.165796 8544 tablet_bootstrap.cc:492] T 8b684222965c468b840a0776167ddb41 P 3a2fba6e705d4e23af93e43abe62f350: No bootstrap required, opened a new log
I20260620 14:48:15.165998 8544 ts_tablet_manager.cc:1397] T 8b684222965c468b840a0776167ddb41 P 3a2fba6e705d4e23af93e43abe62f350: Time spent bootstrapping tablet: real 0.005s user 0.001s sys 0.003s
I20260620 14:48:15.167003 8544 raft_consensus.cc:348] T 8b684222965c468b840a0776167ddb41 P 3a2fba6e705d4e23af93e43abe62f350 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3a2fba6e705d4e23af93e43abe62f350" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 39761 } } peers { permanent_uuid: "a99fedfd9f1844aeb7e6362e68c7d072" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 43441 } } peers { permanent_uuid: "1a858e39015649089fd200123f040125" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 44163 } }
I20260620 14:48:15.167331 8544 raft_consensus.cc:374] T 8b684222965c468b840a0776167ddb41 P 3a2fba6e705d4e23af93e43abe62f350 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 14:48:15.167445 8544 raft_consensus.cc:729] T 8b684222965c468b840a0776167ddb41 P 3a2fba6e705d4e23af93e43abe62f350 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3a2fba6e705d4e23af93e43abe62f350, State: Initialized, Role: FOLLOWER
I20260620 14:48:15.167783 8544 consensus_queue.cc:260] T 8b684222965c468b840a0776167ddb41 P 3a2fba6e705d4e23af93e43abe62f350 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3a2fba6e705d4e23af93e43abe62f350" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 39761 } } peers { permanent_uuid: "a99fedfd9f1844aeb7e6362e68c7d072" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 43441 } } peers { permanent_uuid: "1a858e39015649089fd200123f040125" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 44163 } }
I20260620 14:48:15.168329 8546 tablet_bootstrap.cc:492] T 8b684222965c468b840a0776167ddb41 P a99fedfd9f1844aeb7e6362e68c7d072: Bootstrap starting.
I20260620 14:48:15.168773 8547 tablet_replica.cc:439] T 8b684222965c468b840a0776167ddb41 P 3a2fba6e705d4e23af93e43abe62f350: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3a2fba6e705d4e23af93e43abe62f350" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 39761 } } peers { permanent_uuid: "a99fedfd9f1844aeb7e6362e68c7d072" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 43441 } } peers { permanent_uuid: "1a858e39015649089fd200123f040125" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 44163 } } }
I20260620 14:48:15.169096 8547 tablet_replica.cc:442] T 8b684222965c468b840a0776167ddb41 P 3a2fba6e705d4e23af93e43abe62f350: This TxnStatusTablet replica's current role is: FOLLOWER
I20260620 14:48:15.169308 8544 ts_tablet_manager.cc:1428] T 8b684222965c468b840a0776167ddb41 P 3a2fba6e705d4e23af93e43abe62f350: Time spent starting tablet: real 0.003s user 0.000s sys 0.004s
I20260620 14:48:15.169961 8546 tablet_bootstrap.cc:654] T 8b684222965c468b840a0776167ddb41 P a99fedfd9f1844aeb7e6362e68c7d072: Neither blocks nor log segments found. Creating new log.
I20260620 14:48:15.172013 8546 tablet_bootstrap.cc:492] T 8b684222965c468b840a0776167ddb41 P a99fedfd9f1844aeb7e6362e68c7d072: No bootstrap required, opened a new log
I20260620 14:48:15.172174 8546 ts_tablet_manager.cc:1397] T 8b684222965c468b840a0776167ddb41 P a99fedfd9f1844aeb7e6362e68c7d072: Time spent bootstrapping tablet: real 0.004s user 0.003s sys 0.000s
I20260620 14:48:15.172950 8546 raft_consensus.cc:348] T 8b684222965c468b840a0776167ddb41 P a99fedfd9f1844aeb7e6362e68c7d072 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3a2fba6e705d4e23af93e43abe62f350" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 39761 } } peers { permanent_uuid: "a99fedfd9f1844aeb7e6362e68c7d072" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 43441 } } peers { permanent_uuid: "1a858e39015649089fd200123f040125" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 44163 } }
I20260620 14:48:15.173167 8546 raft_consensus.cc:374] T 8b684222965c468b840a0776167ddb41 P a99fedfd9f1844aeb7e6362e68c7d072 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 14:48:15.173286 8546 raft_consensus.cc:729] T 8b684222965c468b840a0776167ddb41 P a99fedfd9f1844aeb7e6362e68c7d072 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a99fedfd9f1844aeb7e6362e68c7d072, State: Initialized, Role: FOLLOWER
I20260620 14:48:15.173528 8546 consensus_queue.cc:260] T 8b684222965c468b840a0776167ddb41 P a99fedfd9f1844aeb7e6362e68c7d072 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3a2fba6e705d4e23af93e43abe62f350" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 39761 } } peers { permanent_uuid: "a99fedfd9f1844aeb7e6362e68c7d072" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 43441 } } peers { permanent_uuid: "1a858e39015649089fd200123f040125" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 44163 } }
I20260620 14:48:15.174259 8546 ts_tablet_manager.cc:1428] T 8b684222965c468b840a0776167ddb41 P a99fedfd9f1844aeb7e6362e68c7d072: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20260620 14:48:15.174793 8549 tablet_replica.cc:439] T 8b684222965c468b840a0776167ddb41 P a99fedfd9f1844aeb7e6362e68c7d072: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3a2fba6e705d4e23af93e43abe62f350" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 39761 } } peers { permanent_uuid: "a99fedfd9f1844aeb7e6362e68c7d072" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 43441 } } peers { permanent_uuid: "1a858e39015649089fd200123f040125" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 44163 } } }
I20260620 14:48:15.175144 8549 tablet_replica.cc:442] T 8b684222965c468b840a0776167ddb41 P a99fedfd9f1844aeb7e6362e68c7d072: This TxnStatusTablet replica's current role is: FOLLOWER
I20260620 14:48:15.182570 8508 raft_consensus.cc:482] T 8b684222965c468b840a0776167ddb41 P 1a858e39015649089fd200123f040125 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260620 14:48:15.182850 8508 raft_consensus.cc:504] T 8b684222965c468b840a0776167ddb41 P 1a858e39015649089fd200123f040125 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3a2fba6e705d4e23af93e43abe62f350" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 39761 } } peers { permanent_uuid: "a99fedfd9f1844aeb7e6362e68c7d072" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 43441 } } peers { permanent_uuid: "1a858e39015649089fd200123f040125" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 44163 } }
I20260620 14:48:15.183461 8508 leader_election.cc:288] T 8b684222965c468b840a0776167ddb41 P 1a858e39015649089fd200123f040125 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 3a2fba6e705d4e23af93e43abe62f350 (127.0.97.1:39761), a99fedfd9f1844aeb7e6362e68c7d072 (127.0.97.2:43441)
I20260620 14:48:15.184052 8288 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8b684222965c468b840a0776167ddb41" candidate_uuid: "1a858e39015649089fd200123f040125" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a99fedfd9f1844aeb7e6362e68c7d072" is_pre_election: true
I20260620 14:48:15.184056 8153 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8b684222965c468b840a0776167ddb41" candidate_uuid: "1a858e39015649089fd200123f040125" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3a2fba6e705d4e23af93e43abe62f350" is_pre_election: true
I20260620 14:48:15.184331 8288 raft_consensus.cc:2454] T 8b684222965c468b840a0776167ddb41 P a99fedfd9f1844aeb7e6362e68c7d072 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1a858e39015649089fd200123f040125 in term 0.
I20260620 14:48:15.184332 8153 raft_consensus.cc:2454] T 8b684222965c468b840a0776167ddb41 P 3a2fba6e705d4e23af93e43abe62f350 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1a858e39015649089fd200123f040125 in term 0.
I20260620 14:48:15.184748 8360 leader_election.cc:302] T 8b684222965c468b840a0776167ddb41 P 1a858e39015649089fd200123f040125 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1a858e39015649089fd200123f040125, a99fedfd9f1844aeb7e6362e68c7d072; no voters:
I20260620 14:48:15.185058 8508 raft_consensus.cc:2781] T 8b684222965c468b840a0776167ddb41 P 1a858e39015649089fd200123f040125 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260620 14:48:15.185166 8508 raft_consensus.cc:482] T 8b684222965c468b840a0776167ddb41 P 1a858e39015649089fd200123f040125 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260620 14:48:15.185288 8508 raft_consensus.cc:3037] T 8b684222965c468b840a0776167ddb41 P 1a858e39015649089fd200123f040125 [term 0 FOLLOWER]: Advancing to term 1
I20260620 14:48:15.187423 8508 raft_consensus.cc:504] T 8b684222965c468b840a0776167ddb41 P 1a858e39015649089fd200123f040125 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3a2fba6e705d4e23af93e43abe62f350" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 39761 } } peers { permanent_uuid: "a99fedfd9f1844aeb7e6362e68c7d072" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 43441 } } peers { permanent_uuid: "1a858e39015649089fd200123f040125" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 44163 } }
I20260620 14:48:15.188000 8508 leader_election.cc:288] T 8b684222965c468b840a0776167ddb41 P 1a858e39015649089fd200123f040125 [CANDIDATE]: Term 1 election: Requested vote from peers 3a2fba6e705d4e23af93e43abe62f350 (127.0.97.1:39761), a99fedfd9f1844aeb7e6362e68c7d072 (127.0.97.2:43441)
I20260620 14:48:15.188499 8288 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8b684222965c468b840a0776167ddb41" candidate_uuid: "1a858e39015649089fd200123f040125" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a99fedfd9f1844aeb7e6362e68c7d072"
I20260620 14:48:15.188503 8153 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8b684222965c468b840a0776167ddb41" candidate_uuid: "1a858e39015649089fd200123f040125" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3a2fba6e705d4e23af93e43abe62f350"
I20260620 14:48:15.188750 8153 raft_consensus.cc:3037] T 8b684222965c468b840a0776167ddb41 P 3a2fba6e705d4e23af93e43abe62f350 [term 0 FOLLOWER]: Advancing to term 1
I20260620 14:48:15.188756 8288 raft_consensus.cc:3037] T 8b684222965c468b840a0776167ddb41 P a99fedfd9f1844aeb7e6362e68c7d072 [term 0 FOLLOWER]: Advancing to term 1
I20260620 14:48:15.191699 8153 raft_consensus.cc:2454] T 8b684222965c468b840a0776167ddb41 P 3a2fba6e705d4e23af93e43abe62f350 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1a858e39015649089fd200123f040125 in term 1.
I20260620 14:48:15.191699 8288 raft_consensus.cc:2454] T 8b684222965c468b840a0776167ddb41 P a99fedfd9f1844aeb7e6362e68c7d072 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1a858e39015649089fd200123f040125 in term 1.
I20260620 14:48:15.192129 8360 leader_election.cc:302] T 8b684222965c468b840a0776167ddb41 P 1a858e39015649089fd200123f040125 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1a858e39015649089fd200123f040125, a99fedfd9f1844aeb7e6362e68c7d072; no voters:
I20260620 14:48:15.192438 8508 raft_consensus.cc:2781] T 8b684222965c468b840a0776167ddb41 P 1a858e39015649089fd200123f040125 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 14:48:15.192619 8508 raft_consensus.cc:686] T 8b684222965c468b840a0776167ddb41 P 1a858e39015649089fd200123f040125 [term 1 LEADER]: Becoming Leader. State: Replica: 1a858e39015649089fd200123f040125, State: Running, Role: LEADER
I20260620 14:48:15.192857 8508 consensus_queue.cc:237] T 8b684222965c468b840a0776167ddb41 P 1a858e39015649089fd200123f040125 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3a2fba6e705d4e23af93e43abe62f350" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 39761 } } peers { permanent_uuid: "a99fedfd9f1844aeb7e6362e68c7d072" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 43441 } } peers { permanent_uuid: "1a858e39015649089fd200123f040125" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 44163 } }
I20260620 14:48:15.194335 8533 tablet_replica.cc:439] T 8b684222965c468b840a0776167ddb41 P 1a858e39015649089fd200123f040125: TxnStatusTablet state changed. Reason: New leader 1a858e39015649089fd200123f040125. Latest consensus state: current_term: 1 leader_uuid: "1a858e39015649089fd200123f040125" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3a2fba6e705d4e23af93e43abe62f350" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 39761 } } peers { permanent_uuid: "a99fedfd9f1844aeb7e6362e68c7d072" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 43441 } } peers { permanent_uuid: "1a858e39015649089fd200123f040125" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 44163 } } }
I20260620 14:48:15.194702 8533 tablet_replica.cc:442] T 8b684222965c468b840a0776167ddb41 P 1a858e39015649089fd200123f040125: This TxnStatusTablet replica's current role is: LEADER
I20260620 14:48:15.195313 8551 txn_status_manager.cc:873] Waiting until node catch up with all replicated operations in previous term...
I20260620 14:48:15.195537 8551 txn_status_manager.cc:929] Loading transaction status metadata into memory...
I20260620 14:48:15.195936 7991 catalog_manager.cc:5495] T 8b684222965c468b840a0776167ddb41 P 1a858e39015649089fd200123f040125 reported cstate change: term changed from 0 to 1, leader changed from <none> to 1a858e39015649089fd200123f040125 (127.0.97.3). New cstate: current_term: 1 leader_uuid: "1a858e39015649089fd200123f040125" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3a2fba6e705d4e23af93e43abe62f350" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 39761 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "a99fedfd9f1844aeb7e6362e68c7d072" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 43441 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "1a858e39015649089fd200123f040125" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 44163 } health_report { overall_health: HEALTHY } } }
I20260620 14:48:15.214864 8540 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:48:15.208240 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.3:44163 (local address 127.0.0.1:33514)
0620 14:48:15.208486 (+ 246us) negotiation.cc:107] Waiting for socket to connect
0620 14:48:15.208508 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0620 14:48:15.208682 (+ 174us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 14:48:15.209403 (+ 721us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 14:48:15.209407 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 14:48:15.209416 (+ 9us) client_negotiation.cc:189] Negotiated authn=TOKEN
0620 14:48:15.209562 (+ 146us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:48:15.209568 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:48:15.211072 (+ 1504us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:48:15.211075 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:48:15.212129 (+ 1054us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:48:15.212141 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:48:15.213118 (+ 977us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:48:15.213122 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:48:15.213258 (+ 136us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:48:15.213311 (+ 53us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0620 14:48:15.214430 (+ 1119us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0620 14:48:15.214446 (+ 16us) client_negotiation.cc:769] Sending connection context
0620 14:48:15.214512 (+ 66us) client_negotiation.cc:240] Negotiation successful
0620 14:48:15.214631 (+ 119us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":55}
I20260620 14:48:15.215024 8554 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:48:15.208362 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33514 (local address 127.0.97.3:44163)
0620 14:48:15.208972 (+ 610us) server_negotiation.cc:206] Beginning negotiation
0620 14:48:15.208978 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 14:48:15.209002 (+ 24us) server_negotiation.cc:407] Connection header received
0620 14:48:15.209095 (+ 93us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:48:15.209102 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:48:15.209203 (+ 101us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:48:15.209385 (+ 182us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 14:48:15.209690 (+ 305us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:48:15.210923 (+ 1233us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:48:15.212274 (+ 1351us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:48:15.212978 (+ 704us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:48:15.213110 (+ 132us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:48:15.213499 (+ 389us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 14:48:15.214306 (+ 807us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0620 14:48:15.214455 (+ 149us) server_negotiation.cc:1035] Waiting for connection context
0620 14:48:15.214571 (+ 116us) server_negotiation.cc:299] Negotiation successful
0620 14:48:15.214655 (+ 84us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":468,"thread_start_us":334,"threads_started":1}
I20260620 14:48:15.219732 8153 raft_consensus.cc:1261] T 8b684222965c468b840a0776167ddb41 P 3a2fba6e705d4e23af93e43abe62f350 [term 1 FOLLOWER]: Refusing update from remote peer 1a858e39015649089fd200123f040125: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 14:48:15.219868 8288 raft_consensus.cc:1261] T 8b684222965c468b840a0776167ddb41 P a99fedfd9f1844aeb7e6362e68c7d072 [term 1 FOLLOWER]: Refusing update from remote peer 1a858e39015649089fd200123f040125: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 14:48:15.221603 8508 consensus_queue.cc:1035] T 8b684222965c468b840a0776167ddb41 P 1a858e39015649089fd200123f040125 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a99fedfd9f1844aeb7e6362e68c7d072" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 43441 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 14:48:15.221920 8533 consensus_queue.cc:1035] T 8b684222965c468b840a0776167ddb41 P 1a858e39015649089fd200123f040125 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3a2fba6e705d4e23af93e43abe62f350" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 39761 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 14:48:15.226423 8549 tablet_replica.cc:439] T 8b684222965c468b840a0776167ddb41 P a99fedfd9f1844aeb7e6362e68c7d072: TxnStatusTablet state changed. Reason: New leader 1a858e39015649089fd200123f040125. Latest consensus state: current_term: 1 leader_uuid: "1a858e39015649089fd200123f040125" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3a2fba6e705d4e23af93e43abe62f350" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 39761 } } peers { permanent_uuid: "a99fedfd9f1844aeb7e6362e68c7d072" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 43441 } } peers { permanent_uuid: "1a858e39015649089fd200123f040125" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 44163 } } }
I20260620 14:48:15.226819 8549 tablet_replica.cc:442] T 8b684222965c468b840a0776167ddb41 P a99fedfd9f1844aeb7e6362e68c7d072: This TxnStatusTablet replica's current role is: FOLLOWER
I20260620 14:48:15.231067 8555 mvcc.cc:205] Tried to move back new op lower bound from 7298936402811891712 to 7298936402711920640. Current Snapshot: MvccSnapshot[applied={T|T < 7298936402811891712}]
I20260620 14:48:15.234062 8549 tablet_replica.cc:439] T 8b684222965c468b840a0776167ddb41 P a99fedfd9f1844aeb7e6362e68c7d072: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "1a858e39015649089fd200123f040125" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3a2fba6e705d4e23af93e43abe62f350" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 39761 } } peers { permanent_uuid: "a99fedfd9f1844aeb7e6362e68c7d072" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 43441 } } peers { permanent_uuid: "1a858e39015649089fd200123f040125" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 44163 } } }
I20260620 14:48:15.234349 8549 tablet_replica.cc:442] T 8b684222965c468b840a0776167ddb41 P a99fedfd9f1844aeb7e6362e68c7d072: This TxnStatusTablet replica's current role is: FOLLOWER
I20260620 14:48:15.238999 8547 tablet_replica.cc:439] T 8b684222965c468b840a0776167ddb41 P 3a2fba6e705d4e23af93e43abe62f350: TxnStatusTablet state changed. Reason: New leader 1a858e39015649089fd200123f040125. Latest consensus state: current_term: 1 leader_uuid: "1a858e39015649089fd200123f040125" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3a2fba6e705d4e23af93e43abe62f350" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 39761 } } peers { permanent_uuid: "a99fedfd9f1844aeb7e6362e68c7d072" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 43441 } } peers { permanent_uuid: "1a858e39015649089fd200123f040125" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 44163 } } }
I20260620 14:48:15.239362 8547 tablet_replica.cc:442] T 8b684222965c468b840a0776167ddb41 P 3a2fba6e705d4e23af93e43abe62f350: This TxnStatusTablet replica's current role is: FOLLOWER
I20260620 14:48:15.240300 8508 tablet_replica.cc:439] T 8b684222965c468b840a0776167ddb41 P 1a858e39015649089fd200123f040125: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "1a858e39015649089fd200123f040125" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3a2fba6e705d4e23af93e43abe62f350" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 39761 } } peers { permanent_uuid: "a99fedfd9f1844aeb7e6362e68c7d072" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 43441 } } peers { permanent_uuid: "1a858e39015649089fd200123f040125" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 44163 } } }
I20260620 14:48:15.240628 8508 tablet_replica.cc:442] T 8b684222965c468b840a0776167ddb41 P 1a858e39015649089fd200123f040125: This TxnStatusTablet replica's current role is: LEADER
I20260620 14:48:15.243045 8533 tablet_replica.cc:439] T 8b684222965c468b840a0776167ddb41 P 1a858e39015649089fd200123f040125: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "1a858e39015649089fd200123f040125" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3a2fba6e705d4e23af93e43abe62f350" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 39761 } } peers { permanent_uuid: "a99fedfd9f1844aeb7e6362e68c7d072" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 43441 } } peers { permanent_uuid: "1a858e39015649089fd200123f040125" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 44163 } } }
I20260620 14:48:15.243322 8533 tablet_replica.cc:442] T 8b684222965c468b840a0776167ddb41 P 1a858e39015649089fd200123f040125: This TxnStatusTablet replica's current role is: LEADER
I20260620 14:48:15.244688 8547 tablet_replica.cc:439] T 8b684222965c468b840a0776167ddb41 P 3a2fba6e705d4e23af93e43abe62f350: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "1a858e39015649089fd200123f040125" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3a2fba6e705d4e23af93e43abe62f350" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 39761 } } peers { permanent_uuid: "a99fedfd9f1844aeb7e6362e68c7d072" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 43441 } } peers { permanent_uuid: "1a858e39015649089fd200123f040125" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 44163 } } }
I20260620 14:48:15.244925 8547 tablet_replica.cc:442] T 8b684222965c468b840a0776167ddb41 P 3a2fba6e705d4e23af93e43abe62f350: This TxnStatusTablet replica's current role is: FOLLOWER
I20260620 14:48:15.253937 8554 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:48:15.248328 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33520 (local address 127.0.97.3:44163)
0620 14:48:15.248519 (+ 191us) server_negotiation.cc:206] Beginning negotiation
0620 14:48:15.248527 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0620 14:48:15.248767 (+ 240us) server_negotiation.cc:407] Connection header received
0620 14:48:15.248976 (+ 209us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:48:15.248981 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:48:15.249055 (+ 74us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:48:15.249159 (+ 104us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 14:48:15.249639 (+ 480us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:48:15.250491 (+ 852us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:48:15.252085 (+ 1594us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:48:15.252614 (+ 529us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:48:15.252759 (+ 145us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:48:15.253179 (+ 420us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 14:48:15.253283 (+ 104us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0620 14:48:15.253386 (+ 103us) server_negotiation.cc:1035] Waiting for connection context
0620 14:48:15.253542 (+ 156us) server_negotiation.cc:299] Negotiation successful
0620 14:48:15.253619 (+ 77us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":64}
I20260620 14:48:15.270429 8554 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:48:15.264697 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.3:51745 (local address 127.0.97.3:44163)
0620 14:48:15.264878 (+ 181us) server_negotiation.cc:206] Beginning negotiation
0620 14:48:15.264883 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 14:48:15.265347 (+ 464us) server_negotiation.cc:407] Connection header received
0620 14:48:15.265577 (+ 230us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:48:15.265584 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:48:15.265645 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:48:15.265747 (+ 102us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 14:48:15.266291 (+ 544us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:48:15.267052 (+ 761us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:48:15.268582 (+ 1530us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:48:15.268992 (+ 410us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:48:15.269134 (+ 142us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:48:15.269578 (+ 444us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 14:48:15.269679 (+ 101us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0620 14:48:15.269779 (+ 100us) server_negotiation.cc:1035] Waiting for connection context
0620 14:48:15.269922 (+ 143us) server_negotiation.cc:299] Negotiation successful
0620 14:48:15.270042 (+ 120us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":72,"server-negotiator.queue_time_us":61}
I20260620 14:48:15.270505 8567 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:48:15.264603 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.3:44163 (local address 127.0.97.3:51745)
0620 14:48:15.265245 (+ 642us) negotiation.cc:107] Waiting for socket to connect
0620 14:48:15.265263 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0620 14:48:15.265427 (+ 164us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 14:48:15.265771 (+ 344us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 14:48:15.265775 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 14:48:15.265787 (+ 12us) client_negotiation.cc:189] Negotiated authn=TOKEN
0620 14:48:15.266152 (+ 365us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:48:15.266161 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:48:15.267214 (+ 1053us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:48:15.267226 (+ 12us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:48:15.268436 (+ 1210us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:48:15.268448 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:48:15.269181 (+ 733us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:48:15.269189 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:48:15.269349 (+ 160us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:48:15.269404 (+ 55us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0620 14:48:15.269777 (+ 373us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0620 14:48:15.269787 (+ 10us) client_negotiation.cc:769] Sending connection context
0620 14:48:15.269905 (+ 118us) client_negotiation.cc:240] Negotiation successful
0620 14:48:15.270088 (+ 183us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":512,"mutex_wait_us":43,"thread_start_us":396,"threads_started":1}
I20260620 14:48:15.280628 8288 raft_consensus.cc:1261] T f1bb588ed8704523b41ee390ac5f5506 P a99fedfd9f1844aeb7e6362e68c7d072 [term 1 FOLLOWER]: Refusing update from remote peer 1a858e39015649089fd200123f040125: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 14:48:15.280653 8153 raft_consensus.cc:1261] T f1bb588ed8704523b41ee390ac5f5506 P 3a2fba6e705d4e23af93e43abe62f350 [term 1 FOLLOWER]: Refusing update from remote peer 1a858e39015649089fd200123f040125: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 14:48:15.281155 8533 consensus_queue.cc:1035] T f1bb588ed8704523b41ee390ac5f5506 P 1a858e39015649089fd200123f040125 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3a2fba6e705d4e23af93e43abe62f350" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 39761 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 14:48:15.281442 8508 consensus_queue.cc:1035] T f1bb588ed8704523b41ee390ac5f5506 P 1a858e39015649089fd200123f040125 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a99fedfd9f1844aeb7e6362e68c7d072" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 43441 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 14:48:15.349931 7990 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:52204:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 14:48:15.350147 7990 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:52204:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 14:48:15.353549 7990 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 0af6e58150ce4b73a4c412dd6c640a60: Sending DeleteTablet for 3 replicas of tablet f1bb588ed8704523b41ee390ac5f5506
I20260620 14:48:15.354413 8128 tablet_service.cc:1514] Processing DeleteTablet for tablet f1bb588ed8704523b41ee390ac5f5506 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 14:48:15 UTC) from {username='oryx', principal='oryx/127.0.97.62@KRBTEST.COM'} at 127.0.0.1:56560
I20260620 14:48:15.354413 8268 tablet_service.cc:1514] Processing DeleteTablet for tablet f1bb588ed8704523b41ee390ac5f5506 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 14:48:15 UTC) from {username='oryx', principal='oryx/127.0.97.62@KRBTEST.COM'} at 127.0.0.1:36912
I20260620 14:48:15.355077 8573 tablet_replica.cc:330] T f1bb588ed8704523b41ee390ac5f5506 P a99fedfd9f1844aeb7e6362e68c7d072: stopping tablet replica
I20260620 14:48:15.355340 8573 raft_consensus.cc:2229] T f1bb588ed8704523b41ee390ac5f5506 P a99fedfd9f1844aeb7e6362e68c7d072 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 14:48:15.355343 8407 tablet_service.cc:1514] Processing DeleteTablet for tablet f1bb588ed8704523b41ee390ac5f5506 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 14:48:15 UTC) from {username='oryx', principal='oryx/127.0.97.62@KRBTEST.COM'} at 127.0.0.1:33502
I20260620 14:48:15.355722 8573 raft_consensus.cc:2258] T f1bb588ed8704523b41ee390ac5f5506 P a99fedfd9f1844aeb7e6362e68c7d072 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 14:48:15.356925 8573 ts_tablet_manager.cc:1905] T f1bb588ed8704523b41ee390ac5f5506 P a99fedfd9f1844aeb7e6362e68c7d072: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 14:48:15.364377 8573 ts_tablet_manager.cc:1918] T f1bb588ed8704523b41ee390ac5f5506 P a99fedfd9f1844aeb7e6362e68c7d072: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260620 14:48:15.364995 8574 tablet_replica.cc:330] T f1bb588ed8704523b41ee390ac5f5506 P 3a2fba6e705d4e23af93e43abe62f350: stopping tablet replica
I20260620 14:48:15.365319 8575 tablet_replica.cc:330] T f1bb588ed8704523b41ee390ac5f5506 P 1a858e39015649089fd200123f040125: stopping tablet replica
I20260620 14:48:15.365296 8574 raft_consensus.cc:2229] T f1bb588ed8704523b41ee390ac5f5506 P 3a2fba6e705d4e23af93e43abe62f350 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 14:48:15.365562 8575 raft_consensus.cc:2229] T f1bb588ed8704523b41ee390ac5f5506 P 1a858e39015649089fd200123f040125 [term 1 LEADER]: Raft consensus shutting down.
I20260620 14:48:15.365669 8574 raft_consensus.cc:2258] T f1bb588ed8704523b41ee390ac5f5506 P 3a2fba6e705d4e23af93e43abe62f350 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 14:48:15.366235 8575 raft_consensus.cc:2258] T f1bb588ed8704523b41ee390ac5f5506 P 1a858e39015649089fd200123f040125 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 14:48:15.366966 8574 ts_tablet_manager.cc:1905] T f1bb588ed8704523b41ee390ac5f5506 P 3a2fba6e705d4e23af93e43abe62f350: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 14:48:15.367352 8575 ts_tablet_manager.cc:1905] T f1bb588ed8704523b41ee390ac5f5506 P 1a858e39015649089fd200123f040125: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 14:48:15.371454 8573 log.cc:1196] T f1bb588ed8704523b41ee390ac5f5506 P a99fedfd9f1844aeb7e6362e68c7d072: Deleting WAL directory at /tmp/dist-test-taskUci3DU/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781966829091300-388-0/minicluster-data/ts-1/wal/wals/f1bb588ed8704523b41ee390ac5f5506
I20260620 14:48:15.371961 8573 ts_tablet_manager.cc:1939] T f1bb588ed8704523b41ee390ac5f5506 P a99fedfd9f1844aeb7e6362e68c7d072: Deleting consensus metadata
I20260620 14:48:15.373320 7977 catalog_manager.cc:4841] TS a99fedfd9f1844aeb7e6362e68c7d072 (127.0.97.2:43441): tablet f1bb588ed8704523b41ee390ac5f5506 (table test-table [id=7186025b0521402484b97a14988e2514]) successfully deleted
I20260620 14:48:15.375505 8574 ts_tablet_manager.cc:1918] T f1bb588ed8704523b41ee390ac5f5506 P 3a2fba6e705d4e23af93e43abe62f350: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260620 14:48:15.375509 8575 ts_tablet_manager.cc:1918] T f1bb588ed8704523b41ee390ac5f5506 P 1a858e39015649089fd200123f040125: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260620 14:48:15.375665 8575 log.cc:1196] T f1bb588ed8704523b41ee390ac5f5506 P 1a858e39015649089fd200123f040125: Deleting WAL directory at /tmp/dist-test-taskUci3DU/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781966829091300-388-0/minicluster-data/ts-2/wal/wals/f1bb588ed8704523b41ee390ac5f5506
I20260620 14:48:15.375836 8574 log.cc:1196] T f1bb588ed8704523b41ee390ac5f5506 P 3a2fba6e705d4e23af93e43abe62f350: Deleting WAL directory at /tmp/dist-test-taskUci3DU/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781966829091300-388-0/minicluster-data/ts-0/wal/wals/f1bb588ed8704523b41ee390ac5f5506
I20260620 14:48:15.376170 8575 ts_tablet_manager.cc:1939] T f1bb588ed8704523b41ee390ac5f5506 P 1a858e39015649089fd200123f040125: Deleting consensus metadata
I20260620 14:48:15.376437 8574 ts_tablet_manager.cc:1939] T f1bb588ed8704523b41ee390ac5f5506 P 3a2fba6e705d4e23af93e43abe62f350: Deleting consensus metadata
I20260620 14:48:15.377472 7975 catalog_manager.cc:4841] TS 1a858e39015649089fd200123f040125 (127.0.97.3:44163): tablet f1bb588ed8704523b41ee390ac5f5506 (table test-table [id=7186025b0521402484b97a14988e2514]) successfully deleted
I20260620 14:48:15.377556 7976 catalog_manager.cc:4841] TS 3a2fba6e705d4e23af93e43abe62f350 (127.0.97.1:39761): tablet f1bb588ed8704523b41ee390ac5f5506 (table test-table [id=7186025b0521402484b97a14988e2514]) successfully deleted
Jun 20 14:48:15 dist-test-slave-q9j6 krb5kdc[7946](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781966895, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
Jun 20 14:48:15 dist-test-slave-q9j6 krb5kdc[7946](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, test-admin@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
Jun 20 14:48:15 dist-test-slave-q9j6 krb5kdc[7946](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, test-admin@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
W20260620 14:48:15.420351 8541 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 14:48:15.409184 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52226 (local address 127.0.97.62:33203)
0620 14:48:15.409466 (+ 282us) server_negotiation.cc:206] Beginning negotiation
0620 14:48:15.409477 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0620 14:48:15.410033 (+ 556us) server_negotiation.cc:407] Connection header received
0620 14:48:15.410185 (+ 152us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:48:15.410190 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:48:15.410274 (+ 84us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:48:15.410394 (+ 120us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:48:15.411761 (+ 1367us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:48:15.412904 (+ 1143us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:48:15.414168 (+ 1264us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:48:15.414845 (+ 677us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:48:15.414996 (+ 151us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:48:15.420060 (+ 5064us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:52226: BlockingRecv error: recv got EOF from 127.0.0.1:52226 (error 108)
Metrics: {"server-negotiator.queue_time_us":78}
Jun 20 14:48:15 dist-test-slave-q9j6 krb5kdc[7946](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781966895, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM
I20260620 14:48:15.437698 8541 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:48:15.423428 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52236 (local address 127.0.97.62:33203)
0620 14:48:15.423613 (+ 185us) server_negotiation.cc:206] Beginning negotiation
0620 14:48:15.423621 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0620 14:48:15.424049 (+ 428us) server_negotiation.cc:407] Connection header received
0620 14:48:15.424263 (+ 214us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:48:15.424269 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:48:15.424341 (+ 72us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:48:15.424446 (+ 105us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:48:15.425612 (+ 1166us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:48:15.426367 (+ 755us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:48:15.427659 (+ 1292us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:48:15.428150 (+ 491us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:48:15.428305 (+ 155us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:48:15.432065 (+ 3760us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 14:48:15.432071 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 14:48:15.432074 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 14:48:15.432122 (+ 48us) server_negotiation.cc:922] calling sasl_server_start()
0620 14:48:15.434896 (+ 2774us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:48:15.435702 (+ 806us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:48:15.435706 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:48:15.435708 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:48:15.435839 (+ 131us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:48:15.436237 (+ 398us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:48:15.436242 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:48:15.436244 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:48:15.436524 (+ 280us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 14:48:15.436781 (+ 257us) server_negotiation.cc:1035] Waiting for connection context
0620 14:48:15.437087 (+ 306us) server_negotiation.cc:299] Negotiation successful
0620 14:48:15.437326 (+ 239us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":52}
I20260620 14:48:15.441425 7990 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:52236:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260620 14:48:15.442109 7990 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260620 14:48:15.449064 8128 tablet_service.cc:1467] Processing CreateTablet for tablet b2dd3fd422e34fa4aaae16fe9f533092 (DEFAULT_TABLE table=test-table [id=9858508d549b4e3faac27026bce845b5]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 14:48:15.449204 8268 tablet_service.cc:1467] Processing CreateTablet for tablet b2dd3fd422e34fa4aaae16fe9f533092 (DEFAULT_TABLE table=test-table [id=9858508d549b4e3faac27026bce845b5]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 14:48:15.449528 8128 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b2dd3fd422e34fa4aaae16fe9f533092. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 14:48:15.449659 8268 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b2dd3fd422e34fa4aaae16fe9f533092. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 14:48:15.454289 8546 tablet_bootstrap.cc:492] T b2dd3fd422e34fa4aaae16fe9f533092 P a99fedfd9f1844aeb7e6362e68c7d072: Bootstrap starting.
I20260620 14:48:15.454797 8407 tablet_service.cc:1467] Processing CreateTablet for tablet b2dd3fd422e34fa4aaae16fe9f533092 (DEFAULT_TABLE table=test-table [id=9858508d549b4e3faac27026bce845b5]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 14:48:15.455227 8407 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b2dd3fd422e34fa4aaae16fe9f533092. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 14:48:15.455878 8544 tablet_bootstrap.cc:492] T b2dd3fd422e34fa4aaae16fe9f533092 P 3a2fba6e705d4e23af93e43abe62f350: Bootstrap starting.
I20260620 14:48:15.455929 8546 tablet_bootstrap.cc:654] T b2dd3fd422e34fa4aaae16fe9f533092 P a99fedfd9f1844aeb7e6362e68c7d072: Neither blocks nor log segments found. Creating new log.
I20260620 14:48:15.457392 8544 tablet_bootstrap.cc:654] T b2dd3fd422e34fa4aaae16fe9f533092 P 3a2fba6e705d4e23af93e43abe62f350: Neither blocks nor log segments found. Creating new log.
I20260620 14:48:15.464069 8542 tablet_bootstrap.cc:492] T b2dd3fd422e34fa4aaae16fe9f533092 P 1a858e39015649089fd200123f040125: Bootstrap starting.
I20260620 14:48:15.466150 8542 tablet_bootstrap.cc:654] T b2dd3fd422e34fa4aaae16fe9f533092 P 1a858e39015649089fd200123f040125: Neither blocks nor log segments found. Creating new log.
I20260620 14:48:15.468286 8544 tablet_bootstrap.cc:492] T b2dd3fd422e34fa4aaae16fe9f533092 P 3a2fba6e705d4e23af93e43abe62f350: No bootstrap required, opened a new log
I20260620 14:48:15.468474 8544 ts_tablet_manager.cc:1397] T b2dd3fd422e34fa4aaae16fe9f533092 P 3a2fba6e705d4e23af93e43abe62f350: Time spent bootstrapping tablet: real 0.013s user 0.008s sys 0.003s
I20260620 14:48:15.469251 8544 raft_consensus.cc:348] T b2dd3fd422e34fa4aaae16fe9f533092 P 3a2fba6e705d4e23af93e43abe62f350 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3a2fba6e705d4e23af93e43abe62f350" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 39761 } } peers { permanent_uuid: "a99fedfd9f1844aeb7e6362e68c7d072" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 43441 } } peers { permanent_uuid: "1a858e39015649089fd200123f040125" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 44163 } }
I20260620 14:48:15.469529 8544 raft_consensus.cc:374] T b2dd3fd422e34fa4aaae16fe9f533092 P 3a2fba6e705d4e23af93e43abe62f350 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 14:48:15.469596 8544 raft_consensus.cc:729] T b2dd3fd422e34fa4aaae16fe9f533092 P 3a2fba6e705d4e23af93e43abe62f350 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3a2fba6e705d4e23af93e43abe62f350, State: Initialized, Role: FOLLOWER
I20260620 14:48:15.469894 8544 consensus_queue.cc:260] T b2dd3fd422e34fa4aaae16fe9f533092 P 3a2fba6e705d4e23af93e43abe62f350 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3a2fba6e705d4e23af93e43abe62f350" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 39761 } } peers { permanent_uuid: "a99fedfd9f1844aeb7e6362e68c7d072" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 43441 } } peers { permanent_uuid: "1a858e39015649089fd200123f040125" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 44163 } }
I20260620 14:48:15.470443 8544 ts_tablet_manager.cc:1428] T b2dd3fd422e34fa4aaae16fe9f533092 P 3a2fba6e705d4e23af93e43abe62f350: Time spent starting tablet: real 0.002s user 0.000s sys 0.002s
I20260620 14:48:15.476608 8542 tablet_bootstrap.cc:492] T b2dd3fd422e34fa4aaae16fe9f533092 P 1a858e39015649089fd200123f040125: No bootstrap required, opened a new log
I20260620 14:48:15.476830 8542 ts_tablet_manager.cc:1397] T b2dd3fd422e34fa4aaae16fe9f533092 P 1a858e39015649089fd200123f040125: Time spent bootstrapping tablet: real 0.013s user 0.000s sys 0.006s
I20260620 14:48:15.477335 8546 tablet_bootstrap.cc:492] T b2dd3fd422e34fa4aaae16fe9f533092 P a99fedfd9f1844aeb7e6362e68c7d072: No bootstrap required, opened a new log
I20260620 14:48:15.477546 8546 ts_tablet_manager.cc:1397] T b2dd3fd422e34fa4aaae16fe9f533092 P a99fedfd9f1844aeb7e6362e68c7d072: Time spent bootstrapping tablet: real 0.023s user 0.011s sys 0.007s
I20260620 14:48:15.477664 8542 raft_consensus.cc:348] T b2dd3fd422e34fa4aaae16fe9f533092 P 1a858e39015649089fd200123f040125 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3a2fba6e705d4e23af93e43abe62f350" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 39761 } } peers { permanent_uuid: "a99fedfd9f1844aeb7e6362e68c7d072" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 43441 } } peers { permanent_uuid: "1a858e39015649089fd200123f040125" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 44163 } }
I20260620 14:48:15.477962 8542 raft_consensus.cc:374] T b2dd3fd422e34fa4aaae16fe9f533092 P 1a858e39015649089fd200123f040125 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 14:48:15.478078 8542 raft_consensus.cc:729] T b2dd3fd422e34fa4aaae16fe9f533092 P 1a858e39015649089fd200123f040125 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1a858e39015649089fd200123f040125, State: Initialized, Role: FOLLOWER
I20260620 14:48:15.478322 8546 raft_consensus.cc:348] T b2dd3fd422e34fa4aaae16fe9f533092 P a99fedfd9f1844aeb7e6362e68c7d072 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3a2fba6e705d4e23af93e43abe62f350" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 39761 } } peers { permanent_uuid: "a99fedfd9f1844aeb7e6362e68c7d072" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 43441 } } peers { permanent_uuid: "1a858e39015649089fd200123f040125" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 44163 } }
I20260620 14:48:15.478400 8542 consensus_queue.cc:260] T b2dd3fd422e34fa4aaae16fe9f533092 P 1a858e39015649089fd200123f040125 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3a2fba6e705d4e23af93e43abe62f350" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 39761 } } peers { permanent_uuid: "a99fedfd9f1844aeb7e6362e68c7d072" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 43441 } } peers { permanent_uuid: "1a858e39015649089fd200123f040125" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 44163 } }
I20260620 14:48:15.478580 8546 raft_consensus.cc:374] T b2dd3fd422e34fa4aaae16fe9f533092 P a99fedfd9f1844aeb7e6362e68c7d072 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 14:48:15.478823 8546 raft_consensus.cc:729] T b2dd3fd422e34fa4aaae16fe9f533092 P a99fedfd9f1844aeb7e6362e68c7d072 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a99fedfd9f1844aeb7e6362e68c7d072, State: Initialized, Role: FOLLOWER
I20260620 14:48:15.478966 8542 ts_tablet_manager.cc:1428] T b2dd3fd422e34fa4aaae16fe9f533092 P 1a858e39015649089fd200123f040125: Time spent starting tablet: real 0.002s user 0.001s sys 0.001s
I20260620 14:48:15.479117 8546 consensus_queue.cc:260] T b2dd3fd422e34fa4aaae16fe9f533092 P a99fedfd9f1844aeb7e6362e68c7d072 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3a2fba6e705d4e23af93e43abe62f350" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 39761 } } peers { permanent_uuid: "a99fedfd9f1844aeb7e6362e68c7d072" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 43441 } } peers { permanent_uuid: "1a858e39015649089fd200123f040125" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 44163 } }
I20260620 14:48:15.492005 8546 ts_tablet_manager.cc:1428] T b2dd3fd422e34fa4aaae16fe9f533092 P a99fedfd9f1844aeb7e6362e68c7d072: Time spent starting tablet: real 0.014s user 0.002s sys 0.001s
I20260620 14:48:15.699112 8549 raft_consensus.cc:482] T b2dd3fd422e34fa4aaae16fe9f533092 P a99fedfd9f1844aeb7e6362e68c7d072 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260620 14:48:15.699331 8549 raft_consensus.cc:504] T b2dd3fd422e34fa4aaae16fe9f533092 P a99fedfd9f1844aeb7e6362e68c7d072 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3a2fba6e705d4e23af93e43abe62f350" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 39761 } } peers { permanent_uuid: "a99fedfd9f1844aeb7e6362e68c7d072" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 43441 } } peers { permanent_uuid: "1a858e39015649089fd200123f040125" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 44163 } }
I20260620 14:48:15.700479 8549 leader_election.cc:288] T b2dd3fd422e34fa4aaae16fe9f533092 P a99fedfd9f1844aeb7e6362e68c7d072 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 3a2fba6e705d4e23af93e43abe62f350 (127.0.97.1:39761), 1a858e39015649089fd200123f040125 (127.0.97.3:44163)
I20260620 14:48:15.701298 8427 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b2dd3fd422e34fa4aaae16fe9f533092" candidate_uuid: "a99fedfd9f1844aeb7e6362e68c7d072" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1a858e39015649089fd200123f040125" is_pre_election: true
I20260620 14:48:15.701404 8153 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b2dd3fd422e34fa4aaae16fe9f533092" candidate_uuid: "a99fedfd9f1844aeb7e6362e68c7d072" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3a2fba6e705d4e23af93e43abe62f350" is_pre_election: true
I20260620 14:48:15.701578 8427 raft_consensus.cc:2454] T b2dd3fd422e34fa4aaae16fe9f533092 P 1a858e39015649089fd200123f040125 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a99fedfd9f1844aeb7e6362e68c7d072 in term 0.
I20260620 14:48:15.701668 8153 raft_consensus.cc:2454] T b2dd3fd422e34fa4aaae16fe9f533092 P 3a2fba6e705d4e23af93e43abe62f350 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a99fedfd9f1844aeb7e6362e68c7d072 in term 0.
I20260620 14:48:15.702116 8218 leader_election.cc:302] T b2dd3fd422e34fa4aaae16fe9f533092 P a99fedfd9f1844aeb7e6362e68c7d072 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1a858e39015649089fd200123f040125, a99fedfd9f1844aeb7e6362e68c7d072; no voters:
I20260620 14:48:15.702463 8549 raft_consensus.cc:2781] T b2dd3fd422e34fa4aaae16fe9f533092 P a99fedfd9f1844aeb7e6362e68c7d072 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260620 14:48:15.702700 8549 raft_consensus.cc:482] T b2dd3fd422e34fa4aaae16fe9f533092 P a99fedfd9f1844aeb7e6362e68c7d072 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260620 14:48:15.702823 8549 raft_consensus.cc:3037] T b2dd3fd422e34fa4aaae16fe9f533092 P a99fedfd9f1844aeb7e6362e68c7d072 [term 0 FOLLOWER]: Advancing to term 1
I20260620 14:48:15.706226 8549 raft_consensus.cc:504] T b2dd3fd422e34fa4aaae16fe9f533092 P a99fedfd9f1844aeb7e6362e68c7d072 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3a2fba6e705d4e23af93e43abe62f350" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 39761 } } peers { permanent_uuid: "a99fedfd9f1844aeb7e6362e68c7d072" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 43441 } } peers { permanent_uuid: "1a858e39015649089fd200123f040125" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 44163 } }
I20260620 14:48:15.706902 8549 leader_election.cc:288] T b2dd3fd422e34fa4aaae16fe9f533092 P a99fedfd9f1844aeb7e6362e68c7d072 [CANDIDATE]: Term 1 election: Requested vote from peers 3a2fba6e705d4e23af93e43abe62f350 (127.0.97.1:39761), 1a858e39015649089fd200123f040125 (127.0.97.3:44163)
I20260620 14:48:15.707747 8427 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b2dd3fd422e34fa4aaae16fe9f533092" candidate_uuid: "a99fedfd9f1844aeb7e6362e68c7d072" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1a858e39015649089fd200123f040125"
I20260620 14:48:15.707978 8427 raft_consensus.cc:3037] T b2dd3fd422e34fa4aaae16fe9f533092 P 1a858e39015649089fd200123f040125 [term 0 FOLLOWER]: Advancing to term 1
I20260620 14:48:15.711126 8427 raft_consensus.cc:2454] T b2dd3fd422e34fa4aaae16fe9f533092 P 1a858e39015649089fd200123f040125 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a99fedfd9f1844aeb7e6362e68c7d072 in term 1.
I20260620 14:48:15.711637 8153 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b2dd3fd422e34fa4aaae16fe9f533092" candidate_uuid: "a99fedfd9f1844aeb7e6362e68c7d072" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3a2fba6e705d4e23af93e43abe62f350"
I20260620 14:48:15.711781 8218 leader_election.cc:302] T b2dd3fd422e34fa4aaae16fe9f533092 P a99fedfd9f1844aeb7e6362e68c7d072 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1a858e39015649089fd200123f040125, a99fedfd9f1844aeb7e6362e68c7d072; no voters:
I20260620 14:48:15.711864 8153 raft_consensus.cc:3037] T b2dd3fd422e34fa4aaae16fe9f533092 P 3a2fba6e705d4e23af93e43abe62f350 [term 0 FOLLOWER]: Advancing to term 1
I20260620 14:48:15.712092 8549 raft_consensus.cc:2781] T b2dd3fd422e34fa4aaae16fe9f533092 P a99fedfd9f1844aeb7e6362e68c7d072 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 14:48:15.714926 8153 raft_consensus.cc:2454] T b2dd3fd422e34fa4aaae16fe9f533092 P 3a2fba6e705d4e23af93e43abe62f350 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a99fedfd9f1844aeb7e6362e68c7d072 in term 1.
I20260620 14:48:15.721571 8549 raft_consensus.cc:686] T b2dd3fd422e34fa4aaae16fe9f533092 P a99fedfd9f1844aeb7e6362e68c7d072 [term 1 LEADER]: Becoming Leader. State: Replica: a99fedfd9f1844aeb7e6362e68c7d072, State: Running, Role: LEADER
I20260620 14:48:15.722098 8549 consensus_queue.cc:237] T b2dd3fd422e34fa4aaae16fe9f533092 P a99fedfd9f1844aeb7e6362e68c7d072 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3a2fba6e705d4e23af93e43abe62f350" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 39761 } } peers { permanent_uuid: "a99fedfd9f1844aeb7e6362e68c7d072" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 43441 } } peers { permanent_uuid: "1a858e39015649089fd200123f040125" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 44163 } }
I20260620 14:48:15.725476 7991 catalog_manager.cc:5495] T b2dd3fd422e34fa4aaae16fe9f533092 P a99fedfd9f1844aeb7e6362e68c7d072 reported cstate change: term changed from 0 to 1, leader changed from <none> to a99fedfd9f1844aeb7e6362e68c7d072 (127.0.97.2). New cstate: current_term: 1 leader_uuid: "a99fedfd9f1844aeb7e6362e68c7d072" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3a2fba6e705d4e23af93e43abe62f350" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 39761 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "a99fedfd9f1844aeb7e6362e68c7d072" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 43441 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "1a858e39015649089fd200123f040125" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 44163 } health_report { overall_health: UNKNOWN } } }
I20260620 14:48:15.778816 8594 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:48:15.773456 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36922 (local address 127.0.97.2:43441)
0620 14:48:15.774092 (+ 636us) server_negotiation.cc:206] Beginning negotiation
0620 14:48:15.774096 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 14:48:15.774114 (+ 18us) server_negotiation.cc:407] Connection header received
0620 14:48:15.774169 (+ 55us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:48:15.774172 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:48:15.774230 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:48:15.774341 (+ 111us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 14:48:15.774710 (+ 369us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:48:15.775464 (+ 754us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:48:15.776976 (+ 1512us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:48:15.777403 (+ 427us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:48:15.777627 (+ 224us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:48:15.778052 (+ 425us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 14:48:15.778160 (+ 108us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0620 14:48:15.778257 (+ 97us) server_negotiation.cc:1035] Waiting for connection context
0620 14:48:15.778405 (+ 148us) server_negotiation.cc:299] Negotiation successful
0620 14:48:15.778483 (+ 78us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":524,"thread_start_us":384,"threads_started":1}
I20260620 14:48:15.781955 8153 raft_consensus.cc:1261] T b2dd3fd422e34fa4aaae16fe9f533092 P 3a2fba6e705d4e23af93e43abe62f350 [term 1 FOLLOWER]: Refusing update from remote peer a99fedfd9f1844aeb7e6362e68c7d072: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 14:48:15.782124 8427 raft_consensus.cc:1261] T b2dd3fd422e34fa4aaae16fe9f533092 P 1a858e39015649089fd200123f040125 [term 1 FOLLOWER]: Refusing update from remote peer a99fedfd9f1844aeb7e6362e68c7d072: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 14:48:15.782691 8591 consensus_queue.cc:1035] T b2dd3fd422e34fa4aaae16fe9f533092 P a99fedfd9f1844aeb7e6362e68c7d072 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3a2fba6e705d4e23af93e43abe62f350" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 39761 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 14:48:15.783114 8549 consensus_queue.cc:1035] T b2dd3fd422e34fa4aaae16fe9f533092 P a99fedfd9f1844aeb7e6362e68c7d072 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1a858e39015649089fd200123f040125" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 44163 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 14:48:15.794523 7991 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:52236:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 14:48:15.794759 7991 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:52236:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 14:48:15.797892 7991 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 0af6e58150ce4b73a4c412dd6c640a60: Sending DeleteTablet for 3 replicas of tablet b2dd3fd422e34fa4aaae16fe9f533092
I20260620 14:48:15.798746 8128 tablet_service.cc:1514] Processing DeleteTablet for tablet b2dd3fd422e34fa4aaae16fe9f533092 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 14:48:15 UTC) from {username='oryx', principal='oryx/127.0.97.62@KRBTEST.COM'} at 127.0.0.1:56560
I20260620 14:48:15.799049 8574 tablet_replica.cc:330] T b2dd3fd422e34fa4aaae16fe9f533092 P 3a2fba6e705d4e23af93e43abe62f350: stopping tablet replica
I20260620 14:48:15.799139 8407 tablet_service.cc:1514] Processing DeleteTablet for tablet b2dd3fd422e34fa4aaae16fe9f533092 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 14:48:15 UTC) from {username='oryx', principal='oryx/127.0.97.62@KRBTEST.COM'} at 127.0.0.1:33502
I20260620 14:48:15.799444 8575 tablet_replica.cc:330] T b2dd3fd422e34fa4aaae16fe9f533092 P 1a858e39015649089fd200123f040125: stopping tablet replica
I20260620 14:48:15.799710 8575 raft_consensus.cc:2229] T b2dd3fd422e34fa4aaae16fe9f533092 P 1a858e39015649089fd200123f040125 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 14:48:15.799767 8268 tablet_service.cc:1514] Processing DeleteTablet for tablet b2dd3fd422e34fa4aaae16fe9f533092 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 14:48:15 UTC) from {username='oryx', principal='oryx/127.0.97.62@KRBTEST.COM'} at 127.0.0.1:36912
I20260620 14:48:15.799989 8575 raft_consensus.cc:2258] T b2dd3fd422e34fa4aaae16fe9f533092 P 1a858e39015649089fd200123f040125 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 14:48:15.800048 8573 tablet_replica.cc:330] T b2dd3fd422e34fa4aaae16fe9f533092 P a99fedfd9f1844aeb7e6362e68c7d072: stopping tablet replica
I20260620 14:48:15.800249 8573 raft_consensus.cc:2229] T b2dd3fd422e34fa4aaae16fe9f533092 P a99fedfd9f1844aeb7e6362e68c7d072 [term 1 LEADER]: Raft consensus shutting down.
I20260620 14:48:15.800536 8573 raft_consensus.cc:2258] T b2dd3fd422e34fa4aaae16fe9f533092 P a99fedfd9f1844aeb7e6362e68c7d072 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 14:48:15.801765 8573 ts_tablet_manager.cc:1905] T b2dd3fd422e34fa4aaae16fe9f533092 P a99fedfd9f1844aeb7e6362e68c7d072: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 14:48:15.801882 7991 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:52236:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260620 14:48:15.802783 7991 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260620 14:48:15.805356 8574 raft_consensus.cc:2229] T b2dd3fd422e34fa4aaae16fe9f533092 P 3a2fba6e705d4e23af93e43abe62f350 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 14:48:15.805915 8574 pending_rounds.cc:62] T b2dd3fd422e34fa4aaae16fe9f533092 P 3a2fba6e705d4e23af93e43abe62f350: Trying to abort 2 pending ops.
I20260620 14:48:15.806051 8574 pending_rounds.cc:68] T b2dd3fd422e34fa4aaae16fe9f533092 P 3a2fba6e705d4e23af93e43abe62f350: Aborting op as it isn't in flight: id { term: 1 index: 2 } timestamp: 7298936405116592128 op_type: WRITE_OP write_request { tablet_id: "b2dd3fd422e34fa4aaae16fe9f533092" schema { columns { name: "key" type: INT32 is_key: true is_nullable: false immutable: false } columns { name: "val" type: INT32 is_key: false is_nullable: true immutable: false } } row_operations { rows: "<redacted>""\001\003\00090\000\0001\324\000\000" indirect_data: "<redacted>""" } external_consistency_mode: CLIENT_PROPAGATED authz_token { token_data: "\010\333\322\332\321\006\"8\n\ntest-admin\022*\n 9858508d549b4e3faac27026bce845b5\020\001\030\001 \001(\001" signature: "<redacted>"">t\240\267\'\204\026\230\037{\277=\013\262\231\23679Q|\351\335\235i\273\353\320,\006\244\376\377JYy\330\332\213Y\035\220\224\215f\226\273\000\244*,\325v\003\255]i\023\000>\211\323\250d\005" signing_key_seq_num: 0 } } request_id { client_id: "7cde91704d1d4b4db27d85b1108b7758" seq_no: 0 first_incomplete_seq_no: 0 attempt_no: 0 }
I20260620 14:48:15.806711 8574 pending_rounds.cc:68] T b2dd3fd422e34fa4aaae16fe9f533092 P 3a2fba6e705d4e23af93e43abe62f350: Aborting op as it isn't in flight: id { term: 1 index: 1 } timestamp: 7298936404881526784 op_type: NO_OP noop_request { }
I20260620 14:48:15.806859 8574 raft_consensus.cc:2866] T b2dd3fd422e34fa4aaae16fe9f533092 P 3a2fba6e705d4e23af93e43abe62f350 [term 1 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted
I20260620 14:48:15.806977 8574 raft_consensus.cc:2258] T b2dd3fd422e34fa4aaae16fe9f533092 P 3a2fba6e705d4e23af93e43abe62f350 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 14:48:15.810452 8128 tablet_service.cc:1467] Processing CreateTablet for tablet 72f2f311648e4a139da9f024b700b4e1 (DEFAULT_TABLE table=test-table [id=b075a81b93fe403687b150ca3c7f2fb5]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 14:48:15.810886 8128 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 72f2f311648e4a139da9f024b700b4e1. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 14:48:15.810871 8407 tablet_service.cc:1467] Processing CreateTablet for tablet 72f2f311648e4a139da9f024b700b4e1 (DEFAULT_TABLE table=test-table [id=b075a81b93fe403687b150ca3c7f2fb5]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 14:48:15.811319 8407 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 72f2f311648e4a139da9f024b700b4e1. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 14:48:15.812136 8575 ts_tablet_manager.cc:1905] T b2dd3fd422e34fa4aaae16fe9f533092 P 1a858e39015649089fd200123f040125: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 14:48:15.814642 8268 tablet_service.cc:1467] Processing CreateTablet for tablet 72f2f311648e4a139da9f024b700b4e1 (DEFAULT_TABLE table=test-table [id=b075a81b93fe403687b150ca3c7f2fb5]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 14:48:15.814901 8574 ts_tablet_manager.cc:1905] T b2dd3fd422e34fa4aaae16fe9f533092 P 3a2fba6e705d4e23af93e43abe62f350: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 14:48:15.815059 8268 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 72f2f311648e4a139da9f024b700b4e1. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 14:48:15.818651 8575 ts_tablet_manager.cc:1918] T b2dd3fd422e34fa4aaae16fe9f533092 P 1a858e39015649089fd200123f040125: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260620 14:48:15.818811 8575 log.cc:1196] T b2dd3fd422e34fa4aaae16fe9f533092 P 1a858e39015649089fd200123f040125: Deleting WAL directory at /tmp/dist-test-taskUci3DU/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781966829091300-388-0/minicluster-data/ts-2/wal/wals/b2dd3fd422e34fa4aaae16fe9f533092
I20260620 14:48:15.819226 8575 ts_tablet_manager.cc:1939] T b2dd3fd422e34fa4aaae16fe9f533092 P 1a858e39015649089fd200123f040125: Deleting consensus metadata
I20260620 14:48:15.819859 8542 tablet_bootstrap.cc:492] T 72f2f311648e4a139da9f024b700b4e1 P 1a858e39015649089fd200123f040125: Bootstrap starting.
I20260620 14:48:15.821182 8574 ts_tablet_manager.cc:1918] T b2dd3fd422e34fa4aaae16fe9f533092 P 3a2fba6e705d4e23af93e43abe62f350: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260620 14:48:15.821328 8574 log.cc:1196] T b2dd3fd422e34fa4aaae16fe9f533092 P 3a2fba6e705d4e23af93e43abe62f350: Deleting WAL directory at /tmp/dist-test-taskUci3DU/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781966829091300-388-0/minicluster-data/ts-0/wal/wals/b2dd3fd422e34fa4aaae16fe9f533092
I20260620 14:48:15.821584 8542 tablet_bootstrap.cc:654] T 72f2f311648e4a139da9f024b700b4e1 P 1a858e39015649089fd200123f040125: Neither blocks nor log segments found. Creating new log.
I20260620 14:48:15.821767 8574 ts_tablet_manager.cc:1939] T b2dd3fd422e34fa4aaae16fe9f533092 P 3a2fba6e705d4e23af93e43abe62f350: Deleting consensus metadata
I20260620 14:48:15.821755 7975 catalog_manager.cc:4841] TS 1a858e39015649089fd200123f040125 (127.0.97.3:44163): tablet b2dd3fd422e34fa4aaae16fe9f533092 (table test-table [id=9858508d549b4e3faac27026bce845b5]) successfully deleted
I20260620 14:48:15.822942 7976 catalog_manager.cc:4841] TS 3a2fba6e705d4e23af93e43abe62f350 (127.0.97.1:39761): tablet b2dd3fd422e34fa4aaae16fe9f533092 (table test-table [id=9858508d549b4e3faac27026bce845b5]) successfully deleted
I20260620 14:48:15.823683 8573 ts_tablet_manager.cc:1918] T b2dd3fd422e34fa4aaae16fe9f533092 P a99fedfd9f1844aeb7e6362e68c7d072: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260620 14:48:15.823851 8573 log.cc:1196] T b2dd3fd422e34fa4aaae16fe9f533092 P a99fedfd9f1844aeb7e6362e68c7d072: Deleting WAL directory at /tmp/dist-test-taskUci3DU/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781966829091300-388-0/minicluster-data/ts-1/wal/wals/b2dd3fd422e34fa4aaae16fe9f533092
I20260620 14:48:15.824232 8573 ts_tablet_manager.cc:1939] T b2dd3fd422e34fa4aaae16fe9f533092 P a99fedfd9f1844aeb7e6362e68c7d072: Deleting consensus metadata
I20260620 14:48:15.825376 7977 catalog_manager.cc:4841] TS a99fedfd9f1844aeb7e6362e68c7d072 (127.0.97.2:43441): tablet b2dd3fd422e34fa4aaae16fe9f533092 (table test-table [id=9858508d549b4e3faac27026bce845b5]) successfully deleted
I20260620 14:48:15.825505 8544 tablet_bootstrap.cc:492] T 72f2f311648e4a139da9f024b700b4e1 P 3a2fba6e705d4e23af93e43abe62f350: Bootstrap starting.
I20260620 14:48:15.827090 8544 tablet_bootstrap.cc:654] T 72f2f311648e4a139da9f024b700b4e1 P 3a2fba6e705d4e23af93e43abe62f350: Neither blocks nor log segments found. Creating new log.
I20260620 14:48:15.827098 8546 tablet_bootstrap.cc:492] T 72f2f311648e4a139da9f024b700b4e1 P a99fedfd9f1844aeb7e6362e68c7d072: Bootstrap starting.
I20260620 14:48:15.828799 8546 tablet_bootstrap.cc:654] T 72f2f311648e4a139da9f024b700b4e1 P a99fedfd9f1844aeb7e6362e68c7d072: Neither blocks nor log segments found. Creating new log.
I20260620 14:48:15.834475 8542 tablet_bootstrap.cc:492] T 72f2f311648e4a139da9f024b700b4e1 P 1a858e39015649089fd200123f040125: No bootstrap required, opened a new log
I20260620 14:48:15.834717 8542 ts_tablet_manager.cc:1397] T 72f2f311648e4a139da9f024b700b4e1 P 1a858e39015649089fd200123f040125: Time spent bootstrapping tablet: real 0.015s user 0.004s sys 0.004s
I20260620 14:48:15.835831 8542 raft_consensus.cc:348] T 72f2f311648e4a139da9f024b700b4e1 P 1a858e39015649089fd200123f040125 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3a2fba6e705d4e23af93e43abe62f350" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 39761 } } peers { permanent_uuid: "a99fedfd9f1844aeb7e6362e68c7d072" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 43441 } } peers { permanent_uuid: "1a858e39015649089fd200123f040125" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 44163 } }
I20260620 14:48:15.836143 8542 raft_consensus.cc:374] T 72f2f311648e4a139da9f024b700b4e1 P 1a858e39015649089fd200123f040125 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 14:48:15.836258 8542 raft_consensus.cc:729] T 72f2f311648e4a139da9f024b700b4e1 P 1a858e39015649089fd200123f040125 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1a858e39015649089fd200123f040125, State: Initialized, Role: FOLLOWER
I20260620 14:48:15.836539 8542 consensus_queue.cc:260] T 72f2f311648e4a139da9f024b700b4e1 P 1a858e39015649089fd200123f040125 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3a2fba6e705d4e23af93e43abe62f350" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 39761 } } peers { permanent_uuid: "a99fedfd9f1844aeb7e6362e68c7d072" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 43441 } } peers { permanent_uuid: "1a858e39015649089fd200123f040125" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 44163 } }
I20260620 14:48:15.837983 8542 ts_tablet_manager.cc:1428] T 72f2f311648e4a139da9f024b700b4e1 P 1a858e39015649089fd200123f040125: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260620 14:48:15.838179 8546 tablet_bootstrap.cc:492] T 72f2f311648e4a139da9f024b700b4e1 P a99fedfd9f1844aeb7e6362e68c7d072: No bootstrap required, opened a new log
I20260620 14:48:15.838392 8546 ts_tablet_manager.cc:1397] T 72f2f311648e4a139da9f024b700b4e1 P a99fedfd9f1844aeb7e6362e68c7d072: Time spent bootstrapping tablet: real 0.011s user 0.004s sys 0.004s
I20260620 14:48:15.839080 8544 tablet_bootstrap.cc:492] T 72f2f311648e4a139da9f024b700b4e1 P 3a2fba6e705d4e23af93e43abe62f350: No bootstrap required, opened a new log
I20260620 14:48:15.839243 8544 ts_tablet_manager.cc:1397] T 72f2f311648e4a139da9f024b700b4e1 P 3a2fba6e705d4e23af93e43abe62f350: Time spent bootstrapping tablet: real 0.014s user 0.003s sys 0.008s
I20260620 14:48:15.839296 8546 raft_consensus.cc:348] T 72f2f311648e4a139da9f024b700b4e1 P a99fedfd9f1844aeb7e6362e68c7d072 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3a2fba6e705d4e23af93e43abe62f350" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 39761 } } peers { permanent_uuid: "a99fedfd9f1844aeb7e6362e68c7d072" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 43441 } } peers { permanent_uuid: "1a858e39015649089fd200123f040125" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 44163 } }
I20260620 14:48:15.839677 8546 raft_consensus.cc:374] T 72f2f311648e4a139da9f024b700b4e1 P a99fedfd9f1844aeb7e6362e68c7d072 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 14:48:15.839824 8546 raft_consensus.cc:729] T 72f2f311648e4a139da9f024b700b4e1 P a99fedfd9f1844aeb7e6362e68c7d072 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a99fedfd9f1844aeb7e6362e68c7d072, State: Initialized, Role: FOLLOWER
I20260620 14:48:15.840109 8544 raft_consensus.cc:348] T 72f2f311648e4a139da9f024b700b4e1 P 3a2fba6e705d4e23af93e43abe62f350 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3a2fba6e705d4e23af93e43abe62f350" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 39761 } } peers { permanent_uuid: "a99fedfd9f1844aeb7e6362e68c7d072" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 43441 } } peers { permanent_uuid: "1a858e39015649089fd200123f040125" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 44163 } }
I20260620 14:48:15.840116 8546 consensus_queue.cc:260] T 72f2f311648e4a139da9f024b700b4e1 P a99fedfd9f1844aeb7e6362e68c7d072 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3a2fba6e705d4e23af93e43abe62f350" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 39761 } } peers { permanent_uuid: "a99fedfd9f1844aeb7e6362e68c7d072" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 43441 } } peers { permanent_uuid: "1a858e39015649089fd200123f040125" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 44163 } }
I20260620 14:48:15.840329 8544 raft_consensus.cc:374] T 72f2f311648e4a139da9f024b700b4e1 P 3a2fba6e705d4e23af93e43abe62f350 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 14:48:15.840430 8544 raft_consensus.cc:729] T 72f2f311648e4a139da9f024b700b4e1 P 3a2fba6e705d4e23af93e43abe62f350 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3a2fba6e705d4e23af93e43abe62f350, State: Initialized, Role: FOLLOWER
I20260620 14:48:15.840662 8544 consensus_queue.cc:260] T 72f2f311648e4a139da9f024b700b4e1 P 3a2fba6e705d4e23af93e43abe62f350 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3a2fba6e705d4e23af93e43abe62f350" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 39761 } } peers { permanent_uuid: "a99fedfd9f1844aeb7e6362e68c7d072" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 43441 } } peers { permanent_uuid: "1a858e39015649089fd200123f040125" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 44163 } }
I20260620 14:48:15.841219 8544 ts_tablet_manager.cc:1428] T 72f2f311648e4a139da9f024b700b4e1 P 3a2fba6e705d4e23af93e43abe62f350: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20260620 14:48:15.842356 8546 ts_tablet_manager.cc:1428] T 72f2f311648e4a139da9f024b700b4e1 P a99fedfd9f1844aeb7e6362e68c7d072: Time spent starting tablet: real 0.004s user 0.001s sys 0.001s
I20260620 14:48:15.961256 8547 raft_consensus.cc:482] T 72f2f311648e4a139da9f024b700b4e1 P 3a2fba6e705d4e23af93e43abe62f350 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260620 14:48:15.961494 8547 raft_consensus.cc:504] T 72f2f311648e4a139da9f024b700b4e1 P 3a2fba6e705d4e23af93e43abe62f350 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3a2fba6e705d4e23af93e43abe62f350" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 39761 } } peers { permanent_uuid: "a99fedfd9f1844aeb7e6362e68c7d072" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 43441 } } peers { permanent_uuid: "1a858e39015649089fd200123f040125" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 44163 } }
I20260620 14:48:15.963126 8547 leader_election.cc:288] T 72f2f311648e4a139da9f024b700b4e1 P 3a2fba6e705d4e23af93e43abe62f350 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers a99fedfd9f1844aeb7e6362e68c7d072 (127.0.97.2:43441), 1a858e39015649089fd200123f040125 (127.0.97.3:44163)
I20260620 14:48:15.976812 8594 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:48:15.963731 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.1:46681 (local address 127.0.97.2:43441)
0620 14:48:15.963942 (+ 211us) server_negotiation.cc:206] Beginning negotiation
0620 14:48:15.963949 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0620 14:48:15.969357 (+ 5408us) server_negotiation.cc:407] Connection header received
0620 14:48:15.969668 (+ 311us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:48:15.969674 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:48:15.969760 (+ 86us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:48:15.969833 (+ 73us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 14:48:15.970542 (+ 709us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:48:15.971659 (+ 1117us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:48:15.974269 (+ 2610us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:48:15.975519 (+ 1250us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:48:15.975653 (+ 134us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:48:15.975762 (+ 109us) server_negotiation.cc:1035] Waiting for connection context
0620 14:48:15.976384 (+ 622us) server_negotiation.cc:299] Negotiation successful
0620 14:48:15.976533 (+ 149us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":61}
I20260620 14:48:15.977695 8601 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:48:15.963282 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.2:43441 (local address 127.0.97.1:46681)
0620 14:48:15.969218 (+ 5936us) negotiation.cc:107] Waiting for socket to connect
0620 14:48:15.969242 (+ 24us) client_negotiation.cc:174] Beginning negotiation
0620 14:48:15.969474 (+ 232us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 14:48:15.970053 (+ 579us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 14:48:15.970058 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 14:48:15.970088 (+ 30us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 14:48:15.970362 (+ 274us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:48:15.970376 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:48:15.971865 (+ 1489us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:48:15.971870 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:48:15.974050 (+ 2180us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:48:15.974067 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:48:15.975970 (+ 1903us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:48:15.975976 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:48:15.976159 (+ 183us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:48:15.976217 (+ 58us) client_negotiation.cc:769] Sending connection context
0620 14:48:15.977068 (+ 851us) client_negotiation.cc:240] Negotiation successful
0620 14:48:15.977236 (+ 168us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":5762,"threads_started":1}
I20260620 14:48:15.979166 8287 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "72f2f311648e4a139da9f024b700b4e1" candidate_uuid: "3a2fba6e705d4e23af93e43abe62f350" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a99fedfd9f1844aeb7e6362e68c7d072" is_pre_election: true
I20260620 14:48:15.979451 8287 raft_consensus.cc:2454] T 72f2f311648e4a139da9f024b700b4e1 P a99fedfd9f1844aeb7e6362e68c7d072 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3a2fba6e705d4e23af93e43abe62f350 in term 0.
I20260620 14:48:15.979965 8080 leader_election.cc:302] T 72f2f311648e4a139da9f024b700b4e1 P 3a2fba6e705d4e23af93e43abe62f350 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3a2fba6e705d4e23af93e43abe62f350, a99fedfd9f1844aeb7e6362e68c7d072; no voters:
I20260620 14:48:15.980374 8547 raft_consensus.cc:2781] T 72f2f311648e4a139da9f024b700b4e1 P 3a2fba6e705d4e23af93e43abe62f350 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260620 14:48:15.980504 8547 raft_consensus.cc:482] T 72f2f311648e4a139da9f024b700b4e1 P 3a2fba6e705d4e23af93e43abe62f350 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260620 14:48:15.980605 8547 raft_consensus.cc:3037] T 72f2f311648e4a139da9f024b700b4e1 P 3a2fba6e705d4e23af93e43abe62f350 [term 0 FOLLOWER]: Advancing to term 1
I20260620 14:48:15.983933 8547 raft_consensus.cc:504] T 72f2f311648e4a139da9f024b700b4e1 P 3a2fba6e705d4e23af93e43abe62f350 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3a2fba6e705d4e23af93e43abe62f350" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 39761 } } peers { permanent_uuid: "a99fedfd9f1844aeb7e6362e68c7d072" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 43441 } } peers { permanent_uuid: "1a858e39015649089fd200123f040125" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 44163 } }
I20260620 14:48:15.984781 8547 leader_election.cc:288] T 72f2f311648e4a139da9f024b700b4e1 P 3a2fba6e705d4e23af93e43abe62f350 [CANDIDATE]: Term 1 election: Requested vote from peers a99fedfd9f1844aeb7e6362e68c7d072 (127.0.97.2:43441), 1a858e39015649089fd200123f040125 (127.0.97.3:44163)
I20260620 14:48:15.985162 8287 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "72f2f311648e4a139da9f024b700b4e1" candidate_uuid: "3a2fba6e705d4e23af93e43abe62f350" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a99fedfd9f1844aeb7e6362e68c7d072"
I20260620 14:48:15.985379 8287 raft_consensus.cc:3037] T 72f2f311648e4a139da9f024b700b4e1 P a99fedfd9f1844aeb7e6362e68c7d072 [term 0 FOLLOWER]: Advancing to term 1
I20260620 14:48:15.987411 8287 raft_consensus.cc:2454] T 72f2f311648e4a139da9f024b700b4e1 P a99fedfd9f1844aeb7e6362e68c7d072 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3a2fba6e705d4e23af93e43abe62f350 in term 1.
I20260620 14:48:15.988035 8080 leader_election.cc:302] T 72f2f311648e4a139da9f024b700b4e1 P 3a2fba6e705d4e23af93e43abe62f350 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3a2fba6e705d4e23af93e43abe62f350, a99fedfd9f1844aeb7e6362e68c7d072; no voters:
I20260620 14:48:15.988387 8547 raft_consensus.cc:2781] T 72f2f311648e4a139da9f024b700b4e1 P 3a2fba6e705d4e23af93e43abe62f350 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 14:48:15.989321 8602 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:48:15.963847 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.1:55739 (local address 127.0.97.3:44163)
0620 14:48:15.978103 (+ 14256us) server_negotiation.cc:206] Beginning negotiation
0620 14:48:15.978111 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0620 14:48:15.978136 (+ 25us) server_negotiation.cc:407] Connection header received
0620 14:48:15.978372 (+ 236us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:48:15.978378 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:48:15.978454 (+ 76us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:48:15.978542 (+ 88us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 14:48:15.979191 (+ 649us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:48:15.980444 (+ 1253us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:48:15.983591 (+ 3147us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:48:15.987802 (+ 4211us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:48:15.987981 (+ 179us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:48:15.988136 (+ 155us) server_negotiation.cc:1035] Waiting for connection context
0620 14:48:15.988746 (+ 610us) server_negotiation.cc:299] Negotiation successful
0620 14:48:15.988913 (+ 167us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":14090,"thread_start_us":14334,"threads_started":1}
I20260620 14:48:15.989367 8601 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:48:15.963507 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.3:44163 (local address 127.0.97.1:55739)
0620 14:48:15.977947 (+ 14440us) negotiation.cc:107] Waiting for socket to connect
0620 14:48:15.977973 (+ 26us) client_negotiation.cc:174] Beginning negotiation
0620 14:48:15.978124 (+ 151us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 14:48:15.978821 (+ 697us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 14:48:15.978824 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 14:48:15.978846 (+ 22us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 14:48:15.979008 (+ 162us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:48:15.979017 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:48:15.980612 (+ 1595us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:48:15.980620 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:48:15.983353 (+ 2733us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:48:15.983403 (+ 50us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:48:15.988250 (+ 4847us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:48:15.988256 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:48:15.988487 (+ 231us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:48:15.988585 (+ 98us) client_negotiation.cc:769] Sending connection context
0620 14:48:15.988692 (+ 107us) client_negotiation.cc:240] Negotiation successful
0620 14:48:15.988924 (+ 232us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":14335,"thread_start_us":5366,"threads_started":1}
I20260620 14:48:15.989886 8547 raft_consensus.cc:686] T 72f2f311648e4a139da9f024b700b4e1 P 3a2fba6e705d4e23af93e43abe62f350 [term 1 LEADER]: Becoming Leader. State: Replica: 3a2fba6e705d4e23af93e43abe62f350, State: Running, Role: LEADER
I20260620 14:48:15.990259 8426 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "72f2f311648e4a139da9f024b700b4e1" candidate_uuid: "3a2fba6e705d4e23af93e43abe62f350" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1a858e39015649089fd200123f040125" is_pre_election: true
I20260620 14:48:15.990499 8427 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "72f2f311648e4a139da9f024b700b4e1" candidate_uuid: "3a2fba6e705d4e23af93e43abe62f350" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1a858e39015649089fd200123f040125"
I20260620 14:48:15.990602 8426 raft_consensus.cc:2454] T 72f2f311648e4a139da9f024b700b4e1 P 1a858e39015649089fd200123f040125 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3a2fba6e705d4e23af93e43abe62f350 in term 0.
I20260620 14:48:15.991182 8547 consensus_queue.cc:237] T 72f2f311648e4a139da9f024b700b4e1 P 3a2fba6e705d4e23af93e43abe62f350 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3a2fba6e705d4e23af93e43abe62f350" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 39761 } } peers { permanent_uuid: "a99fedfd9f1844aeb7e6362e68c7d072" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 43441 } } peers { permanent_uuid: "1a858e39015649089fd200123f040125" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 44163 } }
I20260620 14:48:15.998076 7991 catalog_manager.cc:5495] T 72f2f311648e4a139da9f024b700b4e1 P 3a2fba6e705d4e23af93e43abe62f350 reported cstate change: term changed from 0 to 1, leader changed from <none> to 3a2fba6e705d4e23af93e43abe62f350 (127.0.97.1). New cstate: current_term: 1 leader_uuid: "3a2fba6e705d4e23af93e43abe62f350" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3a2fba6e705d4e23af93e43abe62f350" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 39761 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "a99fedfd9f1844aeb7e6362e68c7d072" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 43441 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "1a858e39015649089fd200123f040125" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 44163 } health_report { overall_health: UNKNOWN } } }
I20260620 14:48:16.052148 8607 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:48:16.036197 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56568 (local address 127.0.97.1:39761)
0620 14:48:16.045611 (+ 9414us) server_negotiation.cc:206] Beginning negotiation
0620 14:48:16.045617 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 14:48:16.045645 (+ 28us) server_negotiation.cc:407] Connection header received
0620 14:48:16.045771 (+ 126us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:48:16.045776 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:48:16.045883 (+ 107us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:48:16.046763 (+ 880us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 14:48:16.047027 (+ 264us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:48:16.048059 (+ 1032us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:48:16.049527 (+ 1468us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:48:16.049967 (+ 440us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:48:16.050109 (+ 142us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:48:16.050640 (+ 531us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 14:48:16.051375 (+ 735us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0620 14:48:16.051631 (+ 256us) server_negotiation.cc:1035] Waiting for connection context
0620 14:48:16.051691 (+ 60us) server_negotiation.cc:299] Negotiation successful
0620 14:48:16.051770 (+ 79us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":9266,"thread_start_us":11042,"threads_started":1}
I20260620 14:48:16.068872 8602 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:48:16.062579 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.1:59563 (local address 127.0.97.3:44163)
0620 14:48:16.062801 (+ 222us) server_negotiation.cc:206] Beginning negotiation
0620 14:48:16.062807 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 14:48:16.063149 (+ 342us) server_negotiation.cc:407] Connection header received
0620 14:48:16.063367 (+ 218us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:48:16.063371 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:48:16.063432 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:48:16.063512 (+ 80us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 14:48:16.064153 (+ 641us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:48:16.064917 (+ 764us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:48:16.066966 (+ 2049us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:48:16.067441 (+ 475us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:48:16.067541 (+ 100us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:48:16.068186 (+ 645us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 14:48:16.068321 (+ 135us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0620 14:48:16.068402 (+ 81us) server_negotiation.cc:1035] Waiting for connection context
0620 14:48:16.068586 (+ 184us) server_negotiation.cc:299] Negotiation successful
0620 14:48:16.068654 (+ 68us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":93}
I20260620 14:48:16.069136 8611 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:48:16.062369 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.3:44163 (local address 127.0.97.1:59563)
0620 14:48:16.063041 (+ 672us) negotiation.cc:107] Waiting for socket to connect
0620 14:48:16.063062 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0620 14:48:16.063232 (+ 170us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 14:48:16.063576 (+ 344us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 14:48:16.063581 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 14:48:16.063622 (+ 41us) client_negotiation.cc:189] Negotiated authn=TOKEN
0620 14:48:16.063979 (+ 357us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:48:16.063992 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:48:16.065058 (+ 1066us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:48:16.065064 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:48:16.066796 (+ 1732us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:48:16.066813 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:48:16.067642 (+ 829us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:48:16.067649 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:48:16.067847 (+ 198us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:48:16.067968 (+ 121us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0620 14:48:16.068444 (+ 476us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0620 14:48:16.068456 (+ 12us) client_negotiation.cc:769] Sending connection context
0620 14:48:16.068546 (+ 90us) client_negotiation.cc:240] Negotiation successful
0620 14:48:16.068705 (+ 159us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":496,"thread_start_us":318,"threads_started":1}
I20260620 14:48:16.077829 8427 raft_consensus.cc:3037] T 72f2f311648e4a139da9f024b700b4e1 P 1a858e39015649089fd200123f040125 [term 0 FOLLOWER]: Advancing to term 1
I20260620 14:48:16.077932 8287 raft_consensus.cc:1261] T 72f2f311648e4a139da9f024b700b4e1 P a99fedfd9f1844aeb7e6362e68c7d072 [term 1 FOLLOWER]: Refusing update from remote peer 3a2fba6e705d4e23af93e43abe62f350: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 14:48:16.078701 8603 consensus_queue.cc:1035] T 72f2f311648e4a139da9f024b700b4e1 P 3a2fba6e705d4e23af93e43abe62f350 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a99fedfd9f1844aeb7e6362e68c7d072" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 43441 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 14:48:16.081100 8427 raft_consensus.cc:1261] T 72f2f311648e4a139da9f024b700b4e1 P 1a858e39015649089fd200123f040125 [term 1 FOLLOWER]: Refusing update from remote peer 3a2fba6e705d4e23af93e43abe62f350: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 14:48:16.081751 8603 consensus_queue.cc:1035] T 72f2f311648e4a139da9f024b700b4e1 P 3a2fba6e705d4e23af93e43abe62f350 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1a858e39015649089fd200123f040125" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 44163 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 14:48:16.120466 8607 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:48:16.111993 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.3:56077 (local address 127.0.97.1:39761)
0620 14:48:16.112157 (+ 164us) server_negotiation.cc:206] Beginning negotiation
0620 14:48:16.112164 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0620 14:48:16.112786 (+ 622us) server_negotiation.cc:407] Connection header received
0620 14:48:16.113094 (+ 308us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:48:16.113101 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:48:16.113190 (+ 89us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:48:16.113265 (+ 75us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 14:48:16.113856 (+ 591us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:48:16.114565 (+ 709us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:48:16.117841 (+ 3276us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:48:16.118593 (+ 752us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:48:16.118736 (+ 143us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:48:16.119553 (+ 817us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 14:48:16.119728 (+ 175us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0620 14:48:16.119823 (+ 95us) server_negotiation.cc:1035] Waiting for connection context
0620 14:48:16.120047 (+ 224us) server_negotiation.cc:299] Negotiation successful
0620 14:48:16.120139 (+ 92us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":57}
I20260620 14:48:16.120585 8617 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:48:16.111995 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:39761 (local address 127.0.97.3:56077)
0620 14:48:16.112688 (+ 693us) negotiation.cc:107] Waiting for socket to connect
0620 14:48:16.112708 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0620 14:48:16.112899 (+ 191us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 14:48:16.113315 (+ 416us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 14:48:16.113325 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 14:48:16.113338 (+ 13us) client_negotiation.cc:189] Negotiated authn=TOKEN
0620 14:48:16.113680 (+ 342us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:48:16.113693 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:48:16.114707 (+ 1014us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:48:16.114714 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:48:16.116468 (+ 1754us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:48:16.116484 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:48:16.118899 (+ 2415us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:48:16.118909 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:48:16.119189 (+ 280us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:48:16.119312 (+ 123us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0620 14:48:16.119881 (+ 569us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0620 14:48:16.119894 (+ 13us) client_negotiation.cc:769] Sending connection context
0620 14:48:16.119985 (+ 91us) client_negotiation.cc:240] Negotiation successful
0620 14:48:16.120166 (+ 181us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":535,"thread_start_us":379,"threads_started":1}
I20260620 14:48:16.164045 7991 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:52236:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 14:48:16.164317 7991 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:52236:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 14:48:16.167446 7991 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 0af6e58150ce4b73a4c412dd6c640a60: Sending DeleteTablet for 3 replicas of tablet 72f2f311648e4a139da9f024b700b4e1
I20260620 14:48:16.168286 8128 tablet_service.cc:1514] Processing DeleteTablet for tablet 72f2f311648e4a139da9f024b700b4e1 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 14:48:16 UTC) from {username='oryx', principal='oryx/127.0.97.62@KRBTEST.COM'} at 127.0.0.1:56560
I20260620 14:48:16.168363 8268 tablet_service.cc:1514] Processing DeleteTablet for tablet 72f2f311648e4a139da9f024b700b4e1 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 14:48:16 UTC) from {username='oryx', principal='oryx/127.0.97.62@KRBTEST.COM'} at 127.0.0.1:36912
I20260620 14:48:16.168591 8574 tablet_replica.cc:330] T 72f2f311648e4a139da9f024b700b4e1 P 3a2fba6e705d4e23af93e43abe62f350: stopping tablet replica
I20260620 14:48:16.168655 8573 tablet_replica.cc:330] T 72f2f311648e4a139da9f024b700b4e1 P a99fedfd9f1844aeb7e6362e68c7d072: stopping tablet replica
I20260620 14:48:16.168805 8574 raft_consensus.cc:2229] T 72f2f311648e4a139da9f024b700b4e1 P 3a2fba6e705d4e23af93e43abe62f350 [term 1 LEADER]: Raft consensus shutting down.
I20260620 14:48:16.168856 8573 raft_consensus.cc:2229] T 72f2f311648e4a139da9f024b700b4e1 P a99fedfd9f1844aeb7e6362e68c7d072 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 14:48:16.169064 8573 raft_consensus.cc:2258] T 72f2f311648e4a139da9f024b700b4e1 P a99fedfd9f1844aeb7e6362e68c7d072 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 14:48:16.169303 8574 raft_consensus.cc:2258] T 72f2f311648e4a139da9f024b700b4e1 P 3a2fba6e705d4e23af93e43abe62f350 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 14:48:16.170276 8573 ts_tablet_manager.cc:1905] T 72f2f311648e4a139da9f024b700b4e1 P a99fedfd9f1844aeb7e6362e68c7d072: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 14:48:16.170665 8574 ts_tablet_manager.cc:1905] T 72f2f311648e4a139da9f024b700b4e1 P 3a2fba6e705d4e23af93e43abe62f350: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 14:48:16.174774 8573 ts_tablet_manager.cc:1918] T 72f2f311648e4a139da9f024b700b4e1 P a99fedfd9f1844aeb7e6362e68c7d072: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260620 14:48:16.174934 8573 log.cc:1196] T 72f2f311648e4a139da9f024b700b4e1 P a99fedfd9f1844aeb7e6362e68c7d072: Deleting WAL directory at /tmp/dist-test-taskUci3DU/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781966829091300-388-0/minicluster-data/ts-1/wal/wals/72f2f311648e4a139da9f024b700b4e1
I20260620 14:48:16.175734 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskUci3DU/build/asan/bin/kudu pid 8062
I20260620 14:48:16.175493 8573 ts_tablet_manager.cc:1939] T 72f2f311648e4a139da9f024b700b4e1 P a99fedfd9f1844aeb7e6362e68c7d072: Deleting consensus metadata
I20260620 14:48:16.177996 8407 tablet_service.cc:1514] Processing DeleteTablet for tablet 72f2f311648e4a139da9f024b700b4e1 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 14:48:16 UTC) from {username='oryx', principal='oryx/127.0.97.62@KRBTEST.COM'} at 127.0.0.1:33502
I20260620 14:48:16.178293 8575 tablet_replica.cc:330] T 72f2f311648e4a139da9f024b700b4e1 P 1a858e39015649089fd200123f040125: stopping tablet replica
I20260620 14:48:16.178478 8575 raft_consensus.cc:2229] T 72f2f311648e4a139da9f024b700b4e1 P 1a858e39015649089fd200123f040125 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 14:48:16.178685 8575 raft_consensus.cc:2258] T 72f2f311648e4a139da9f024b700b4e1 P 1a858e39015649089fd200123f040125 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 14:48:16.178920 7977 catalog_manager.cc:4841] TS a99fedfd9f1844aeb7e6362e68c7d072 (127.0.97.2:43441): tablet 72f2f311648e4a139da9f024b700b4e1 (table test-table [id=b075a81b93fe403687b150ca3c7f2fb5]) successfully deleted
I20260620 14:48:16.178965 8574 ts_tablet_manager.cc:1918] T 72f2f311648e4a139da9f024b700b4e1 P 3a2fba6e705d4e23af93e43abe62f350: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260620 14:48:16.179112 8574 log.cc:1196] T 72f2f311648e4a139da9f024b700b4e1 P 3a2fba6e705d4e23af93e43abe62f350: Deleting WAL directory at /tmp/dist-test-taskUci3DU/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781966829091300-388-0/minicluster-data/ts-0/wal/wals/72f2f311648e4a139da9f024b700b4e1
I20260620 14:48:16.179535 8574 ts_tablet_manager.cc:1939] T 72f2f311648e4a139da9f024b700b4e1 P 3a2fba6e705d4e23af93e43abe62f350: Deleting consensus metadata
I20260620 14:48:16.180291 8575 ts_tablet_manager.cc:1905] T 72f2f311648e4a139da9f024b700b4e1 P 1a858e39015649089fd200123f040125: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 14:48:16.180617 7976 catalog_manager.cc:4841] TS 3a2fba6e705d4e23af93e43abe62f350 (127.0.97.1:39761): tablet 72f2f311648e4a139da9f024b700b4e1 (table test-table [id=b075a81b93fe403687b150ca3c7f2fb5]) successfully deleted
I20260620 14:48:16.189610 8575 ts_tablet_manager.cc:1918] T 72f2f311648e4a139da9f024b700b4e1 P 1a858e39015649089fd200123f040125: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260620 14:48:16.189779 8575 log.cc:1196] T 72f2f311648e4a139da9f024b700b4e1 P 1a858e39015649089fd200123f040125: Deleting WAL directory at /tmp/dist-test-taskUci3DU/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781966829091300-388-0/minicluster-data/ts-2/wal/wals/72f2f311648e4a139da9f024b700b4e1
I20260620 14:48:16.190275 8575 ts_tablet_manager.cc:1939] T 72f2f311648e4a139da9f024b700b4e1 P 1a858e39015649089fd200123f040125: Deleting consensus metadata
I20260620 14:48:16.191419 7975 catalog_manager.cc:4841] TS 1a858e39015649089fd200123f040125 (127.0.97.3:44163): tablet 72f2f311648e4a139da9f024b700b4e1 (table test-table [id=b075a81b93fe403687b150ca3c7f2fb5]) successfully deleted
Jun 20 14:48:16 dist-test-slave-q9j6 krb5kdc[7946](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781966895, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.0.97.1@KRBTEST.COM
I20260620 14:48:16.205386 8607 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:48:16.177114 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56580 (local address 127.0.97.1:39761)
0620 14:48:16.177306 (+ 192us) server_negotiation.cc:206] Beginning negotiation
0620 14:48:16.177312 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 14:48:16.188327 (+ 11015us) server_negotiation.cc:407] Connection header received
0620 14:48:16.188644 (+ 317us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:48:16.188650 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:48:16.188732 (+ 82us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:48:16.188862 (+ 130us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:48:16.190479 (+ 1617us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:48:16.191639 (+ 1160us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:48:16.193426 (+ 1787us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:48:16.193879 (+ 453us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:48:16.193973 (+ 94us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:48:16.198225 (+ 4252us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 14:48:16.198243 (+ 18us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 14:48:16.198271 (+ 28us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 14:48:16.198325 (+ 54us) server_negotiation.cc:922] calling sasl_server_start()
0620 14:48:16.202419 (+ 4094us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:48:16.203130 (+ 711us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:48:16.203140 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:48:16.203144 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:48:16.203319 (+ 175us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:48:16.203705 (+ 386us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:48:16.203711 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:48:16.203714 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:48:16.204172 (+ 458us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 14:48:16.204549 (+ 377us) server_negotiation.cc:1035] Waiting for connection context
0620 14:48:16.204726 (+ 177us) server_negotiation.cc:299] Negotiation successful
0620 14:48:16.204979 (+ 253us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":62}
I20260620 14:48:16.206107 8193 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 14:48:16.321755 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskUci3DU/build/asan/bin/kudu with pid 8062
I20260620 14:48:16.342093 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskUci3DU/build/asan/bin/kudu pid 8202
Jun 20 14:48:16 dist-test-slave-q9j6 krb5kdc[7946](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781966895, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.0.97.2@KRBTEST.COM
I20260620 14:48:16.357352 8594 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:48:16.342795 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36938 (local address 127.0.97.2:43441)
0620 14:48:16.342990 (+ 195us) server_negotiation.cc:206] Beginning negotiation
0620 14:48:16.343002 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0620 14:48:16.343031 (+ 29us) server_negotiation.cc:407] Connection header received
0620 14:48:16.343260 (+ 229us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:48:16.343272 (+ 12us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:48:16.343334 (+ 62us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:48:16.343436 (+ 102us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:48:16.344732 (+ 1296us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:48:16.345439 (+ 707us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:48:16.346670 (+ 1231us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:48:16.347077 (+ 407us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:48:16.347216 (+ 139us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:48:16.351522 (+ 4306us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 14:48:16.351538 (+ 16us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 14:48:16.351556 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 14:48:16.351661 (+ 105us) server_negotiation.cc:922] calling sasl_server_start()
0620 14:48:16.354496 (+ 2835us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:48:16.355194 (+ 698us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:48:16.355209 (+ 15us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:48:16.355212 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:48:16.355349 (+ 137us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:48:16.355799 (+ 450us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:48:16.355804 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:48:16.355814 (+ 10us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:48:16.356168 (+ 354us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 14:48:16.356482 (+ 314us) server_negotiation.cc:1035] Waiting for connection context
0620 14:48:16.356811 (+ 329us) server_negotiation.cc:299] Negotiation successful
0620 14:48:16.357036 (+ 225us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":78}
I20260620 14:48:16.358007 8333 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 14:48:16.479243 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskUci3DU/build/asan/bin/kudu with pid 8202
I20260620 14:48:16.501554 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskUci3DU/build/asan/bin/kudu pid 8342
Jun 20 14:48:16 dist-test-slave-q9j6 krb5kdc[7946](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781966895, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.0.97.3@KRBTEST.COM
I20260620 14:48:16.517089 8602 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:48:16.502393 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33534 (local address 127.0.97.3:44163)
0620 14:48:16.502582 (+ 189us) server_negotiation.cc:206] Beginning negotiation
0620 14:48:16.502588 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 14:48:16.502608 (+ 20us) server_negotiation.cc:407] Connection header received
0620 14:48:16.502696 (+ 88us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:48:16.502702 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:48:16.502784 (+ 82us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:48:16.502896 (+ 112us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:48:16.504248 (+ 1352us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:48:16.504970 (+ 722us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:48:16.506206 (+ 1236us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:48:16.506600 (+ 394us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:48:16.506748 (+ 148us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:48:16.511376 (+ 4628us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 14:48:16.511391 (+ 15us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 14:48:16.511407 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 14:48:16.511447 (+ 40us) server_negotiation.cc:922] calling sasl_server_start()
0620 14:48:16.514159 (+ 2712us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:48:16.514886 (+ 727us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:48:16.514895 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:48:16.514897 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:48:16.515033 (+ 136us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:48:16.515430 (+ 397us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:48:16.515435 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:48:16.515438 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:48:16.515878 (+ 440us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 14:48:16.516174 (+ 296us) server_negotiation.cc:1035] Waiting for connection context
0620 14:48:16.516504 (+ 330us) server_negotiation.cc:299] Negotiation successful
0620 14:48:16.516728 (+ 224us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":75}
I20260620 14:48:16.517776 8472 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 14:48:16.666813 8624 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:48:16.665825 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:39761 (local address 127.0.97.3:44689)
0620 14:48:16.666428 (+ 603us) negotiation.cc:107] Waiting for socket to connect
0620 14:48:16.666564 (+ 136us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.1:39761: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":491,"thread_start_us":388,"threads_started":1}
I20260620 14:48:16.666879 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskUci3DU/build/asan/bin/kudu with pid 8342
I20260620 14:48:16.692719 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskUci3DU/build/asan/bin/kudu pid 7959
I20260620 14:48:16.693930 8031 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 14:48:16.813778 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskUci3DU/build/asan/bin/kudu with pid 7959
2026-06-20T14:48:16Z chronyd exiting
[ OK ] SecurityITest.TestNonDefaultPrincipal (6277 ms)
[ RUN ] SecurityITest.TestNonExistentPrincipal
Loading random data
Initializing database '/tmp/dist-test-taskUci3DU/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1781966829091300-388-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 20 14:48:16 dist-test-slave-q9j6 krb5kdc[8628](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 20 14:48:16 dist-test-slave-q9j6 krb5kdc[8628](info): set up 2 sockets
Jun 20 14:48:16 dist-test-slave-q9j6 krb5kdc[8628](info): commencing operation
krb5kdc: starting...
W20260620 14:48:17.908578 388 mini_kdc.cc:121] Time spent starting KDC: real 1.049s user 0.005s sys 0.017s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskUci3DU/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1781966829091300-388-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskUci3DU/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1781966829091300-388-0/krb5kdc/test-user.keytab.
Jun 20 14:48:18 dist-test-slave-q9j6 krb5kdc[8628](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781966898, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-20T14:48:18Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T14:48:18Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.97.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskUci3DU/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1781966829091300-388-0/krb5kdc/kudu_127.0.97.62.keytab.
Entry for principal kudu/127.0.97.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskUci3DU/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1781966829091300-388-0/krb5kdc/kudu_127.0.97.62.keytab.
I20260620 14:48:18.131146 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskUci3DU/build/asan/bin/kudu
/tmp/dist-test-taskUci3DU/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskUci3DU/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1781966829091300-388-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskUci3DU/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1781966829091300-388-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskUci3DU/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1781966829091300-388-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskUci3DU/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1781966829091300-388-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:42271
--webserver_interface=127.0.97.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskUci3DU/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1781966829091300-388-0/krb5kdc/kudu_127.0.97.62.keytab
--principal=kudu/127.0.97.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:40413
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.97.62:42271
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--principal=oryx with env {KRB5CCNAME=/tmp/dist-test-taskUci3DU/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1781966829091300-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskUci3DU/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1781966829091300-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskUci3DU/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1781966829091300-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 14:48:18.458989 8641 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 14:48:18.459384 8641 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 14:48:18.459539 8641 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 14:48:18.466859 8641 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 14:48:18.467031 8641 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 14:48:18.467144 8641 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 14:48:18.467249 8641 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 14:48:18.467351 8641 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 14:48:18.477352 8641 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:40413
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskUci3DU/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1781966829091300-388-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskUci3DU/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1781966829091300-388-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.97.62:42271
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskUci3DU/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1781966829091300-388-0/krb5kdc/kudu_127.0.97.62.keytab
--principal=oryx
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:42271
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskUci3DU/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1781966829091300-388-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.97.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskUci3DU/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1781966829091300-388-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 13:41:20 UTC on ccycloud.quasar-vdagoi.root.comops.site
build id 4457462
ASAN enabled
I20260620 14:48:18.478762 8641 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 14:48:18.480211 8641 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 14:48:18.492115 8646 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 14:48:18.492180 8647 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 14:48:18.492208 8649 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 14:48:18.493041 8641 server_base.cc:1034] running on GCE node
I20260620 14:48:18.493956 8641 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 14:48:18.495397 8641 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 14:48:18.496608 8641 hybrid_clock.cc:648] HybridClock initialized: now 1781966898496547 us; error 61 us; skew 500 ppm
I20260620 14:48:18.496929 8641 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Runtime error: kinit failed: unable to login from keytab: Keytab contains no suitable keys for oryx@KRBTEST.COM
2026-06-20T14:48:18Z chronyd exiting
[ OK ] SecurityITest.TestNonExistentPrincipal (1753 ms)
[ RUN ] SecurityITest.TestMismatchingPrincipals
Loading random data
Initializing database '/tmp/dist-test-taskUci3DU/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781966829091300-388-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 20 14:48:18 dist-test-slave-q9j6 krb5kdc[8655](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 20 14:48:18 dist-test-slave-q9j6 krb5kdc[8655](info): set up 2 sockets
Jun 20 14:48:18 dist-test-slave-q9j6 krb5kdc[8655](info): commencing operation
krb5kdc: starting...
W20260620 14:48:19.664681 388 mini_kdc.cc:121] Time spent starting KDC: real 1.051s user 0.000s sys 0.024s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskUci3DU/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781966829091300-388-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskUci3DU/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781966829091300-388-0/krb5kdc/test-user.keytab.
Jun 20 14:48:19 dist-test-slave-q9j6 krb5kdc[8655](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781966899, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-20T14:48:19Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T14:48:19Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.97.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskUci3DU/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781966829091300-388-0/krb5kdc/kudu_127.0.97.62.keytab.
Entry for principal kudu/127.0.97.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskUci3DU/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781966829091300-388-0/krb5kdc/kudu_127.0.97.62.keytab.
I20260620 14:48:19.890882 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskUci3DU/build/asan/bin/kudu
/tmp/dist-test-taskUci3DU/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskUci3DU/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781966829091300-388-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskUci3DU/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781966829091300-388-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskUci3DU/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781966829091300-388-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskUci3DU/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781966829091300-388-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:41725
--webserver_interface=127.0.97.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskUci3DU/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781966829091300-388-0/krb5kdc/kudu_127.0.97.62.keytab
--principal=kudu/127.0.97.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:39945
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.97.62:41725
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskUci3DU/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781966829091300-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskUci3DU/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781966829091300-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskUci3DU/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781966829091300-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 14:48:20.227061 8668 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 14:48:20.227368 8668 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 14:48:20.227543 8668 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 14:48:20.234858 8668 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 14:48:20.234975 8668 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 14:48:20.235021 8668 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 14:48:20.235112 8668 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 14:48:20.235183 8668 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 14:48:20.245090 8668 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:39945
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskUci3DU/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781966829091300-388-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskUci3DU/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781966829091300-388-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.97.62:41725
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskUci3DU/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781966829091300-388-0/krb5kdc/kudu_127.0.97.62.keytab
--principal=kudu/127.0.97.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:41725
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskUci3DU/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781966829091300-388-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.97.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskUci3DU/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781966829091300-388-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 13:41:20 UTC on ccycloud.quasar-vdagoi.root.comops.site
build id 4457462
ASAN enabled
I20260620 14:48:20.246493 8668 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 14:48:20.247920 8668 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 14:48:20.259940 8673 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 14:48:20.259953 8674 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 14:48:20.259975 8676 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 14:48:20.260860 8668 server_base.cc:1034] running on GCE node
I20260620 14:48:20.261706 8668 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 14:48:20.263197 8668 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 14:48:20.264433 8668 hybrid_clock.cc:648] HybridClock initialized: now 1781966900264393 us; error 60 us; skew 500 ppm
I20260620 14:48:20.264801 8668 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 14:48:20 dist-test-slave-q9j6 krb5kdc[8655](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781966900, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 14:48:20.271371 8668 init.cc:377] Logged in from keytab as kudu/127.0.97.62@KRBTEST.COM (short username kudu)
I20260620 14:48:20.273121 8668 webserver.cc:460] Webserver started at http://127.0.97.62:42213/ using document root <none> and password file <none>
I20260620 14:48:20.273885 8668 fs_manager.cc:359] Metadata directory not provided
I20260620 14:48:20.274034 8668 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 14:48:20.274371 8668 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 14:48:20.277525 8668 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskUci3DU/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781966829091300-388-0/minicluster-data/master-0/data/instance:
uuid: "2538e6ad3f224be8952920801072ebb3"
format_stamp: "Formatted at 2026-06-20 14:48:20 on dist-test-slave-q9j6"
server_key: "0b3ccd5bc5ffcc7e0ea503d3052c6903"
server_key_iv: "5ad5aa4993a98fdddab83bd19532e05b"
server_key_version: "encryptionkey@0"
I20260620 14:48:20.278363 8668 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskUci3DU/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781966829091300-388-0/minicluster-data/master-0/wal/instance:
uuid: "2538e6ad3f224be8952920801072ebb3"
format_stamp: "Formatted at 2026-06-20 14:48:20 on dist-test-slave-q9j6"
server_key: "0b3ccd5bc5ffcc7e0ea503d3052c6903"
server_key_iv: "5ad5aa4993a98fdddab83bd19532e05b"
server_key_version: "encryptionkey@0"
I20260620 14:48:20.284042 8668 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.002s sys 0.005s
I20260620 14:48:20.287966 8683 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 14:48:20.289463 8668 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260620 14:48:20.289667 8668 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskUci3DU/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781966829091300-388-0/minicluster-data/master-0/data,/tmp/dist-test-taskUci3DU/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781966829091300-388-0/minicluster-data/master-0/wal
uuid: "2538e6ad3f224be8952920801072ebb3"
format_stamp: "Formatted at 2026-06-20 14:48:20 on dist-test-slave-q9j6"
server_key: "0b3ccd5bc5ffcc7e0ea503d3052c6903"
server_key_iv: "5ad5aa4993a98fdddab83bd19532e05b"
server_key_version: "encryptionkey@0"
I20260620 14:48:20.289896 8668 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskUci3DU/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781966829091300-388-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskUci3DU/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781966829091300-388-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskUci3DU/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781966829091300-388-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 14:48:20.327946 8668 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 14:48:20.332165 8668 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 14:48:20.332481 8668 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 14:48:20.351359 8668 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.62:41725
I20260620 14:48:20.351383 8734 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.62:41725 every 8 connection(s)
I20260620 14:48:20.352878 8668 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskUci3DU/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781966829091300-388-0/minicluster-data/master-0/data/info.pb
I20260620 14:48:20.354045 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskUci3DU/build/asan/bin/kudu as pid 8668
I20260620 14:48:20.354283 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskUci3DU/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781966829091300-388-0/minicluster-data/master-0/wal/instance
I20260620 14:48:20.354915 388 external_mini_cluster.cc:1428] Setting key 2116e771efd5e654248f29f92f064329
I20260620 14:48:20.357396 8735 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
Jun 20 14:48:20 dist-test-slave-q9j6 krb5kdc[8655](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781966899, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260620 14:48:20.371099 8735 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 2538e6ad3f224be8952920801072ebb3: Bootstrap starting.
I20260620 14:48:20.374748 8735 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 2538e6ad3f224be8952920801072ebb3: Neither blocks nor log segments found. Creating new log.
I20260620 14:48:20.375936 8735 log.cc:824] T 00000000000000000000000000000000 P 2538e6ad3f224be8952920801072ebb3: Log is configured to *not* fsync() on all Append() calls
I20260620 14:48:20.379323 8738 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:48:20.357017 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58918 (local address 127.0.97.62:41725)
0620 14:48:20.357877 (+ 860us) server_negotiation.cc:206] Beginning negotiation
0620 14:48:20.357889 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0620 14:48:20.357930 (+ 41us) server_negotiation.cc:407] Connection header received
0620 14:48:20.359573 (+ 1643us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:48:20.359652 (+ 79us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:48:20.360117 (+ 465us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:48:20.360500 (+ 383us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:48:20.362374 (+ 1874us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:48:20.363902 (+ 1528us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:48:20.365610 (+ 1708us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:48:20.366304 (+ 694us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:48:20.366490 (+ 186us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:48:20.371105 (+ 4615us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 14:48:20.371114 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 14:48:20.371133 (+ 19us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 14:48:20.371182 (+ 49us) server_negotiation.cc:922] calling sasl_server_start()
0620 14:48:20.374390 (+ 3208us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:48:20.375260 (+ 870us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:48:20.375274 (+ 14us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:48:20.375282 (+ 8us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:48:20.375461 (+ 179us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:48:20.375901 (+ 440us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:48:20.375906 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:48:20.375909 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:48:20.376651 (+ 742us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 14:48:20.376989 (+ 338us) server_negotiation.cc:1035] Waiting for connection context
0620 14:48:20.377548 (+ 559us) server_negotiation.cc:299] Negotiation successful
0620 14:48:20.377988 (+ 440us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":644,"thread_start_us":407,"threads_started":1}
I20260620 14:48:20.379442 8735 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 2538e6ad3f224be8952920801072ebb3: No bootstrap required, opened a new log
I20260620 14:48:20.387406 8735 raft_consensus.cc:348] T 00000000000000000000000000000000 P 2538e6ad3f224be8952920801072ebb3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2538e6ad3f224be8952920801072ebb3" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 41725 } }
I20260620 14:48:20.387949 8735 raft_consensus.cc:374] T 00000000000000000000000000000000 P 2538e6ad3f224be8952920801072ebb3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 14:48:20.388062 8735 raft_consensus.cc:729] T 00000000000000000000000000000000 P 2538e6ad3f224be8952920801072ebb3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2538e6ad3f224be8952920801072ebb3, State: Initialized, Role: FOLLOWER
I20260620 14:48:20.388697 8735 consensus_queue.cc:260] T 00000000000000000000000000000000 P 2538e6ad3f224be8952920801072ebb3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2538e6ad3f224be8952920801072ebb3" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 41725 } }
I20260620 14:48:20.388943 8735 raft_consensus.cc:388] T 00000000000000000000000000000000 P 2538e6ad3f224be8952920801072ebb3 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 14:48:20.389092 8735 raft_consensus.cc:482] T 00000000000000000000000000000000 P 2538e6ad3f224be8952920801072ebb3 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 14:48:20.389278 8735 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 2538e6ad3f224be8952920801072ebb3 [term 0 FOLLOWER]: Advancing to term 1
I20260620 14:48:20.391444 8735 raft_consensus.cc:504] T 00000000000000000000000000000000 P 2538e6ad3f224be8952920801072ebb3 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2538e6ad3f224be8952920801072ebb3" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 41725 } }
I20260620 14:48:20.391984 8735 leader_election.cc:302] T 00000000000000000000000000000000 P 2538e6ad3f224be8952920801072ebb3 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 2538e6ad3f224be8952920801072ebb3; no voters:
I20260620 14:48:20.392599 8735 leader_election.cc:288] T 00000000000000000000000000000000 P 2538e6ad3f224be8952920801072ebb3 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260620 14:48:20.392747 8740 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 2538e6ad3f224be8952920801072ebb3 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 14:48:20.394148 8740 raft_consensus.cc:686] T 00000000000000000000000000000000 P 2538e6ad3f224be8952920801072ebb3 [term 1 LEADER]: Becoming Leader. State: Replica: 2538e6ad3f224be8952920801072ebb3, State: Running, Role: LEADER
I20260620 14:48:20.394770 8740 consensus_queue.cc:237] T 00000000000000000000000000000000 P 2538e6ad3f224be8952920801072ebb3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2538e6ad3f224be8952920801072ebb3" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 41725 } }
I20260620 14:48:20.394909 8735 sys_catalog.cc:564] T 00000000000000000000000000000000 P 2538e6ad3f224be8952920801072ebb3 [sys.catalog]: configured and running, proceeding with master startup.
I20260620 14:48:20.398329 8741 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2538e6ad3f224be8952920801072ebb3 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "2538e6ad3f224be8952920801072ebb3" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2538e6ad3f224be8952920801072ebb3" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 41725 } } }
I20260620 14:48:20.398893 8741 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2538e6ad3f224be8952920801072ebb3 [sys.catalog]: This master's current role is: LEADER
I20260620 14:48:20.398300 8742 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2538e6ad3f224be8952920801072ebb3 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 2538e6ad3f224be8952920801072ebb3. Latest consensus state: current_term: 1 leader_uuid: "2538e6ad3f224be8952920801072ebb3" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2538e6ad3f224be8952920801072ebb3" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 41725 } } }
I20260620 14:48:20.399075 8742 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2538e6ad3f224be8952920801072ebb3 [sys.catalog]: This master's current role is: LEADER
I20260620 14:48:20.402864 8749 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 14:48:20.407181 8749 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 14:48:20.416253 8749 catalog_manager.cc:1347] Generated new cluster ID: efc1fcfe5d0c408684a4e9a7af42f424
I20260620 14:48:20.416405 8749 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 14:48:20.455487 8749 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 14:48:20.457444 8749 catalog_manager.cc:1504] Loading token signing keys...
I20260620 14:48:20.469199 8749 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 2538e6ad3f224be8952920801072ebb3: Generated new TSK 0
I20260620 14:48:20.470131 8749 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.0.97.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskUci3DU/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781966829091300-388-0/krb5kdc/kudu_127.0.97.1.keytab.
Entry for principal kudu/127.0.97.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskUci3DU/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781966829091300-388-0/krb5kdc/kudu_127.0.97.1.keytab.
I20260620 14:48:20.530521 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskUci3DU/build/asan/bin/kudu
/tmp/dist-test-taskUci3DU/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskUci3DU/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781966829091300-388-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskUci3DU/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781966829091300-388-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskUci3DU/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781966829091300-388-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskUci3DU/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781966829091300-388-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.1:0
--local_ip_for_outbound_sockets=127.0.97.1
--webserver_interface=127.0.97.1
--webserver_port=0
--tserver_master_addrs=127.0.97.62:41725
--keytab_file=/tmp/dist-test-taskUci3DU/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781966829091300-388-0/krb5kdc/kudu_127.0.97.1.keytab
--principal=kudu/127.0.97.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:39945
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskUci3DU/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781966829091300-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskUci3DU/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781966829091300-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskUci3DU/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781966829091300-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 14:48:20.872526 8761 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 14:48:20.872835 8761 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 14:48:20.873006 8761 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 14:48:20.880421 8761 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 14:48:20.880545 8761 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 14:48:20.880762 8761 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.1
I20260620 14:48:20.890913 8761 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:39945
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskUci3DU/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781966829091300-388-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskUci3DU/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781966829091300-388-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskUci3DU/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781966829091300-388-0/krb5kdc/kudu_127.0.97.1.keytab
--principal=kudu/127.0.97.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskUci3DU/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781966829091300-388-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.97.1
--webserver_port=0
--tserver_master_addrs=127.0.97.62:41725
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.1
--log_dir=/tmp/dist-test-taskUci3DU/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781966829091300-388-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 13:41:20 UTC on ccycloud.quasar-vdagoi.root.comops.site
build id 4457462
ASAN enabled
I20260620 14:48:20.892378 8761 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 14:48:20.893795 8761 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 14:48:20.907819 8767 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 14:48:20.907886 8766 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 14:48:20.907930 8769 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 14:48:20.908533 8761 server_base.cc:1034] running on GCE node
I20260620 14:48:20.909165 8761 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 14:48:20.910174 8761 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 14:48:20.911420 8761 hybrid_clock.cc:648] HybridClock initialized: now 1781966900911350 us; error 67 us; skew 500 ppm
I20260620 14:48:20.911877 8761 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 14:48:20 dist-test-slave-q9j6 krb5kdc[8655](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781966900, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 14:48:20.915196 8761 init.cc:377] Logged in from keytab as kudu/127.0.97.1@KRBTEST.COM (short username kudu)
I20260620 14:48:20.916841 8761 webserver.cc:460] Webserver started at http://127.0.97.1:36279/ using document root <none> and password file <none>
I20260620 14:48:20.917584 8761 fs_manager.cc:359] Metadata directory not provided
I20260620 14:48:20.917737 8761 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 14:48:20.918057 8761 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 14:48:20.921159 8761 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskUci3DU/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781966829091300-388-0/minicluster-data/ts-0/data/instance:
uuid: "3e929367ca2a4cbd8f573833779da5a5"
format_stamp: "Formatted at 2026-06-20 14:48:20 on dist-test-slave-q9j6"
server_key: "25bf6cf522b1578294962344ecb80e47"
server_key_iv: "36be690accabd96a85693fe246465faa"
server_key_version: "encryptionkey@0"
I20260620 14:48:20.922013 8761 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskUci3DU/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781966829091300-388-0/minicluster-data/ts-0/wal/instance:
uuid: "3e929367ca2a4cbd8f573833779da5a5"
format_stamp: "Formatted at 2026-06-20 14:48:20 on dist-test-slave-q9j6"
server_key: "25bf6cf522b1578294962344ecb80e47"
server_key_iv: "36be690accabd96a85693fe246465faa"
server_key_version: "encryptionkey@0"
I20260620 14:48:20.927528 8761 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.007s sys 0.000s
I20260620 14:48:20.931327 8776 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 14:48:20.932950 8761 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260620 14:48:20.933161 8761 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskUci3DU/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781966829091300-388-0/minicluster-data/ts-0/data,/tmp/dist-test-taskUci3DU/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781966829091300-388-0/minicluster-data/ts-0/wal
uuid: "3e929367ca2a4cbd8f573833779da5a5"
format_stamp: "Formatted at 2026-06-20 14:48:20 on dist-test-slave-q9j6"
server_key: "25bf6cf522b1578294962344ecb80e47"
server_key_iv: "36be690accabd96a85693fe246465faa"
server_key_version: "encryptionkey@0"
I20260620 14:48:20.933395 8761 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskUci3DU/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781966829091300-388-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskUci3DU/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781966829091300-388-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskUci3DU/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781966829091300-388-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 14:48:20.952153 8761 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 14:48:20.956363 8761 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 14:48:20.956672 8761 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 14:48:20.957643 8761 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 14:48:20.959021 8761 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 14:48:20.959138 8761 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 14:48:20.959295 8761 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 14:48:20.959368 8761 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 14:48:20.993492 8761 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.1:33589
I20260620 14:48:20.993535 8888 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.1:33589 every 8 connection(s)
I20260620 14:48:20.995014 8761 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskUci3DU/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781966829091300-388-0/minicluster-data/ts-0/data/info.pb
I20260620 14:48:21.003474 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskUci3DU/build/asan/bin/kudu as pid 8761
I20260620 14:48:21.003719 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskUci3DU/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781966829091300-388-0/minicluster-data/ts-0/wal/instance
I20260620 14:48:21.004451 388 external_mini_cluster.cc:1428] Setting key 0f9546df089b7da8bebc096ec692246d
Jun 20 14:48:21 dist-test-slave-q9j6 krb5kdc[8655](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781966900, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.1@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260620 14:48:21.018014 8891 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:48:20.997622 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.1:56813 (local address 127.0.97.62:41725)
0620 14:48:20.998057 (+ 435us) server_negotiation.cc:206] Beginning negotiation
0620 14:48:20.998062 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 14:48:20.999179 (+ 1117us) server_negotiation.cc:407] Connection header received
0620 14:48:21.001166 (+ 1987us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:48:21.001170 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:48:21.001241 (+ 71us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:48:21.001344 (+ 103us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:48:21.002939 (+ 1595us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:48:21.004102 (+ 1163us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:48:21.006140 (+ 2038us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:48:21.006736 (+ 596us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:48:21.006870 (+ 134us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:48:21.011077 (+ 4207us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 14:48:21.011087 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 14:48:21.011095 (+ 8us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 14:48:21.011144 (+ 49us) server_negotiation.cc:922] calling sasl_server_start()
0620 14:48:21.014529 (+ 3385us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:48:21.015549 (+ 1020us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:48:21.015554 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:48:21.015556 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:48:21.015743 (+ 187us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:48:21.016253 (+ 510us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:48:21.016256 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:48:21.016265 (+ 9us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:48:21.016518 (+ 253us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 14:48:21.016619 (+ 101us) server_negotiation.cc:1035] Waiting for connection context
0620 14:48:21.017586 (+ 967us) server_negotiation.cc:299] Negotiation successful
0620 14:48:21.017760 (+ 174us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":327,"thread_start_us":226,"threads_started":1}
I20260620 14:48:21.019758 8892 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:48:20.998143 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:41725 (local address 127.0.97.1:56813)
0620 14:48:20.998997 (+ 854us) negotiation.cc:107] Waiting for socket to connect
0620 14:48:20.999053 (+ 56us) client_negotiation.cc:174] Beginning negotiation
0620 14:48:21.000882 (+ 1829us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 14:48:21.001586 (+ 704us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 14:48:21.001595 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 14:48:21.002152 (+ 557us) client_negotiation.cc:189] Negotiated authn=SASL
0620 14:48:21.002766 (+ 614us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:48:21.002784 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:48:21.004268 (+ 1484us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:48:21.004273 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:48:21.005938 (+ 1665us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:48:21.005952 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:48:21.006959 (+ 1007us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:48:21.006964 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:48:21.007164 (+ 200us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:48:21.007900 (+ 736us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 14:48:21.007942 (+ 42us) client_negotiation.cc:656] calling sasl_client_start()
0620 14:48:21.010801 (+ 2859us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 14:48:21.014704 (+ 3903us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 14:48:21.014714 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 14:48:21.014733 (+ 19us) client_negotiation.cc:760] calling sasl_client_step()
0620 14:48:21.015403 (+ 670us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 14:48:21.015861 (+ 458us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 14:48:21.015869 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 14:48:21.015875 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0620 14:48:21.016133 (+ 258us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 14:48:21.016651 (+ 518us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 14:48:21.016663 (+ 12us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 14:48:21.017155 (+ 492us) client_negotiation.cc:769] Sending connection context
0620 14:48:21.017518 (+ 363us) client_negotiation.cc:240] Negotiation successful
0620 14:48:21.018130 (+ 612us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":550,"thread_start_us":353,"threads_started":1}
I20260620 14:48:21.021476 8889 heartbeater.cc:347] Connected to a master server at 127.0.97.62:41725
I20260620 14:48:21.021826 8889 heartbeater.cc:464] Registering TS with master...
I20260620 14:48:21.022779 8889 heartbeater.cc:511] Master 127.0.97.62:41725 requested a full tablet report, sending...
I20260620 14:48:21.025115 8700 ts_manager.cc:194] Registered new tserver with Master: 3e929367ca2a4cbd8f573833779da5a5 (127.0.97.1:33589)
I20260620 14:48:21.026561 8700 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.97.1@KRBTEST.COM'} at 127.0.97.1:56813
WARNING: no policy specified for kudu/127.0.97.2@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskUci3DU/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781966829091300-388-0/krb5kdc/kudu_127.0.97.2.keytab.
Entry for principal kudu/127.0.97.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskUci3DU/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781966829091300-388-0/krb5kdc/kudu_127.0.97.2.keytab.
I20260620 14:48:21.057150 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskUci3DU/build/asan/bin/kudu
/tmp/dist-test-taskUci3DU/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskUci3DU/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781966829091300-388-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskUci3DU/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781966829091300-388-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskUci3DU/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781966829091300-388-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskUci3DU/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781966829091300-388-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.2:0
--local_ip_for_outbound_sockets=127.0.97.2
--webserver_interface=127.0.97.2
--webserver_port=0
--tserver_master_addrs=127.0.97.62:41725
--keytab_file=/tmp/dist-test-taskUci3DU/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781966829091300-388-0/krb5kdc/kudu_127.0.97.2.keytab
--principal=kudu/127.0.97.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:39945
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskUci3DU/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781966829091300-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskUci3DU/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781966829091300-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskUci3DU/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781966829091300-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 14:48:21.400558 8895 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 14:48:21.400851 8895 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 14:48:21.400954 8895 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 14:48:21.408424 8895 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 14:48:21.408545 8895 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 14:48:21.408667 8895 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.2
I20260620 14:48:21.418711 8895 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:39945
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskUci3DU/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781966829091300-388-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskUci3DU/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781966829091300-388-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskUci3DU/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781966829091300-388-0/krb5kdc/kudu_127.0.97.2.keytab
--principal=kudu/127.0.97.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskUci3DU/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781966829091300-388-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.97.2
--webserver_port=0
--tserver_master_addrs=127.0.97.62:41725
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.2
--log_dir=/tmp/dist-test-taskUci3DU/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781966829091300-388-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 13:41:20 UTC on ccycloud.quasar-vdagoi.root.comops.site
build id 4457462
ASAN enabled
I20260620 14:48:21.420094 8895 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 14:48:21.421370 8895 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 14:48:21.435425 8900 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 14:48:21.435365 8903 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 14:48:21.435451 8901 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 14:48:21.436553 8895 server_base.cc:1034] running on GCE node
I20260620 14:48:21.437295 8895 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 14:48:21.438437 8895 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 14:48:21.439698 8895 hybrid_clock.cc:648] HybridClock initialized: now 1781966901439642 us; error 55 us; skew 500 ppm
I20260620 14:48:21.440035 8895 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 14:48:21 dist-test-slave-q9j6 krb5kdc[8655](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781966901, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 14:48:21.443157 8895 init.cc:377] Logged in from keytab as kudu/127.0.97.2@KRBTEST.COM (short username kudu)
I20260620 14:48:21.444649 8895 webserver.cc:460] Webserver started at http://127.0.97.2:41337/ using document root <none> and password file <none>
I20260620 14:48:21.445401 8895 fs_manager.cc:359] Metadata directory not provided
I20260620 14:48:21.445551 8895 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 14:48:21.445871 8895 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 14:48:21.448990 8895 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskUci3DU/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781966829091300-388-0/minicluster-data/ts-1/data/instance:
uuid: "4d756e74cf814ef184391921f8c2bb54"
format_stamp: "Formatted at 2026-06-20 14:48:21 on dist-test-slave-q9j6"
server_key: "080abc3796168454037b682c3493a12d"
server_key_iv: "d8a19159c631ff0a89703c12235b190a"
server_key_version: "encryptionkey@0"
I20260620 14:48:21.449854 8895 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskUci3DU/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781966829091300-388-0/minicluster-data/ts-1/wal/instance:
uuid: "4d756e74cf814ef184391921f8c2bb54"
format_stamp: "Formatted at 2026-06-20 14:48:21 on dist-test-slave-q9j6"
server_key: "080abc3796168454037b682c3493a12d"
server_key_iv: "d8a19159c631ff0a89703c12235b190a"
server_key_version: "encryptionkey@0"
I20260620 14:48:21.455379 8895 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.007s sys 0.000s
I20260620 14:48:21.459141 8910 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 14:48:21.460682 8895 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260620 14:48:21.460882 8895 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskUci3DU/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781966829091300-388-0/minicluster-data/ts-1/data,/tmp/dist-test-taskUci3DU/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781966829091300-388-0/minicluster-data/ts-1/wal
uuid: "4d756e74cf814ef184391921f8c2bb54"
format_stamp: "Formatted at 2026-06-20 14:48:21 on dist-test-slave-q9j6"
server_key: "080abc3796168454037b682c3493a12d"
server_key_iv: "d8a19159c631ff0a89703c12235b190a"
server_key_version: "encryptionkey@0"
I20260620 14:48:21.461115 8895 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskUci3DU/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781966829091300-388-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskUci3DU/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781966829091300-388-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskUci3DU/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781966829091300-388-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 14:48:21.478647 8895 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 14:48:21.482666 8895 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 14:48:21.482990 8895 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 14:48:21.484023 8895 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 14:48:21.485410 8895 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 14:48:21.485518 8895 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 14:48:21.485666 8895 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 14:48:21.485736 8895 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 14:48:21.519111 8895 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.2:40895
I20260620 14:48:21.519155 9022 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.2:40895 every 8 connection(s)
I20260620 14:48:21.520593 8895 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskUci3DU/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781966829091300-388-0/minicluster-data/ts-1/data/info.pb
I20260620 14:48:21.529980 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskUci3DU/build/asan/bin/kudu as pid 8895
I20260620 14:48:21.530197 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskUci3DU/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781966829091300-388-0/minicluster-data/ts-1/wal/instance
I20260620 14:48:21.530726 388 external_mini_cluster.cc:1428] Setting key 2220961dbc3cae7e295142061eb98b07
Jun 20 14:48:21 dist-test-slave-q9j6 krb5kdc[8655](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781966901, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.2@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260620 14:48:21.542027 9025 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:48:21.523171 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.2:55989 (local address 127.0.97.62:41725)
0620 14:48:21.523639 (+ 468us) server_negotiation.cc:206] Beginning negotiation
0620 14:48:21.523645 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 14:48:21.524675 (+ 1030us) server_negotiation.cc:407] Connection header received
0620 14:48:21.526427 (+ 1752us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:48:21.526432 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:48:21.526502 (+ 70us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:48:21.526599 (+ 97us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:48:21.528169 (+ 1570us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:48:21.528885 (+ 716us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:48:21.530535 (+ 1650us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:48:21.531202 (+ 667us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:48:21.531348 (+ 146us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:48:21.535086 (+ 3738us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 14:48:21.535095 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 14:48:21.535103 (+ 8us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 14:48:21.535150 (+ 47us) server_negotiation.cc:922] calling sasl_server_start()
0620 14:48:21.538796 (+ 3646us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:48:21.539636 (+ 840us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:48:21.539645 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:48:21.539652 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:48:21.539819 (+ 167us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:48:21.540269 (+ 450us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:48:21.540273 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:48:21.540276 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:48:21.540547 (+ 271us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 14:48:21.540702 (+ 155us) server_negotiation.cc:1035] Waiting for connection context
0620 14:48:21.541417 (+ 715us) server_negotiation.cc:299] Negotiation successful
0620 14:48:21.541640 (+ 223us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":321,"thread_start_us":212,"threads_started":1}
I20260620 14:48:21.543112 9026 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:48:21.523793 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:41725 (local address 127.0.97.2:55989)
0620 14:48:21.524496 (+ 703us) negotiation.cc:107] Waiting for socket to connect
0620 14:48:21.524541 (+ 45us) client_negotiation.cc:174] Beginning negotiation
0620 14:48:21.526160 (+ 1619us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 14:48:21.526815 (+ 655us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 14:48:21.526824 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 14:48:21.527345 (+ 521us) client_negotiation.cc:189] Negotiated authn=SASL
0620 14:48:21.528011 (+ 666us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:48:21.528029 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:48:21.529017 (+ 988us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:48:21.529020 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:48:21.530382 (+ 1362us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:48:21.530395 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:48:21.531485 (+ 1090us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:48:21.531494 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:48:21.531782 (+ 288us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:48:21.532527 (+ 745us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 14:48:21.532573 (+ 46us) client_negotiation.cc:656] calling sasl_client_start()
0620 14:48:21.534826 (+ 2253us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 14:48:21.538957 (+ 4131us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 14:48:21.538965 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 14:48:21.538982 (+ 17us) client_negotiation.cc:760] calling sasl_client_step()
0620 14:48:21.539448 (+ 466us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 14:48:21.539932 (+ 484us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 14:48:21.539936 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 14:48:21.539938 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 14:48:21.540131 (+ 193us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 14:48:21.540681 (+ 550us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 14:48:21.540690 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 14:48:21.541056 (+ 366us) client_negotiation.cc:769] Sending connection context
0620 14:48:21.541333 (+ 277us) client_negotiation.cc:240] Negotiation successful
0620 14:48:21.541898 (+ 565us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":464,"thread_start_us":277,"threads_started":1}
I20260620 14:48:21.544788 9023 heartbeater.cc:347] Connected to a master server at 127.0.97.62:41725
I20260620 14:48:21.545111 9023 heartbeater.cc:464] Registering TS with master...
I20260620 14:48:21.545861 9023 heartbeater.cc:511] Master 127.0.97.62:41725 requested a full tablet report, sending...
I20260620 14:48:21.547492 8700 ts_manager.cc:194] Registered new tserver with Master: 4d756e74cf814ef184391921f8c2bb54 (127.0.97.2:40895)
I20260620 14:48:21.548827 8700 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.97.2@KRBTEST.COM'} at 127.0.97.2:55989
WARNING: no policy specified for kudu/127.0.97.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskUci3DU/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781966829091300-388-0/krb5kdc/kudu_127.0.97.3.keytab.
Entry for principal kudu/127.0.97.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskUci3DU/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781966829091300-388-0/krb5kdc/kudu_127.0.97.3.keytab.
I20260620 14:48:21.580901 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskUci3DU/build/asan/bin/kudu
/tmp/dist-test-taskUci3DU/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskUci3DU/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781966829091300-388-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskUci3DU/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781966829091300-388-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskUci3DU/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781966829091300-388-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskUci3DU/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781966829091300-388-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.3:0
--local_ip_for_outbound_sockets=127.0.97.3
--webserver_interface=127.0.97.3
--webserver_port=0
--tserver_master_addrs=127.0.97.62:41725
--keytab_file=/tmp/dist-test-taskUci3DU/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781966829091300-388-0/krb5kdc/kudu_127.0.97.3.keytab
--principal=kudu/127.0.97.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:39945
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskUci3DU/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781966829091300-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskUci3DU/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781966829091300-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskUci3DU/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781966829091300-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 14:48:21.914415 9029 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 14:48:21.914721 9029 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 14:48:21.914816 9029 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 14:48:21.922209 9029 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 14:48:21.922329 9029 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 14:48:21.922446 9029 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.3
I20260620 14:48:21.932413 9029 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:39945
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskUci3DU/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781966829091300-388-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskUci3DU/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781966829091300-388-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskUci3DU/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781966829091300-388-0/krb5kdc/kudu_127.0.97.3.keytab
--principal=kudu/127.0.97.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskUci3DU/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781966829091300-388-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.97.3
--webserver_port=0
--tserver_master_addrs=127.0.97.62:41725
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.3
--log_dir=/tmp/dist-test-taskUci3DU/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781966829091300-388-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 13:41:20 UTC on ccycloud.quasar-vdagoi.root.comops.site
build id 4457462
ASAN enabled
I20260620 14:48:21.933763 9029 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 14:48:21.935094 9029 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 14:48:21.948509 9034 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 14:48:21.948499 9035 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 14:48:21.948637 9037 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 14:48:21.949357 9029 server_base.cc:1034] running on GCE node
I20260620 14:48:21.949997 9029 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 14:48:21.951009 9029 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 14:48:21.952235 9029 hybrid_clock.cc:648] HybridClock initialized: now 1781966901952169 us; error 69 us; skew 500 ppm
I20260620 14:48:21.952574 9029 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 14:48:21 dist-test-slave-q9j6 krb5kdc[8655](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781966901, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 14:48:21.955812 9029 init.cc:377] Logged in from keytab as kudu/127.0.97.3@KRBTEST.COM (short username kudu)
I20260620 14:48:21.957350 9029 webserver.cc:460] Webserver started at http://127.0.97.3:39895/ using document root <none> and password file <none>
I20260620 14:48:21.958055 9029 fs_manager.cc:359] Metadata directory not provided
I20260620 14:48:21.958165 9029 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 14:48:21.958463 9029 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 14:48:21.961589 9029 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskUci3DU/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781966829091300-388-0/minicluster-data/ts-2/data/instance:
uuid: "5f650bd720844ca18d8ffda594914597"
format_stamp: "Formatted at 2026-06-20 14:48:21 on dist-test-slave-q9j6"
server_key: "4a16d24b8fa670f973b16484efa3e3d2"
server_key_iv: "423c0babded941b7e6fd7fad47566dff"
server_key_version: "encryptionkey@0"
I20260620 14:48:21.962467 9029 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskUci3DU/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781966829091300-388-0/minicluster-data/ts-2/wal/instance:
uuid: "5f650bd720844ca18d8ffda594914597"
format_stamp: "Formatted at 2026-06-20 14:48:21 on dist-test-slave-q9j6"
server_key: "4a16d24b8fa670f973b16484efa3e3d2"
server_key_iv: "423c0babded941b7e6fd7fad47566dff"
server_key_version: "encryptionkey@0"
I20260620 14:48:21.968063 9029 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.002s sys 0.005s
I20260620 14:48:21.971697 9044 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 14:48:21.973205 9029 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260620 14:48:21.973387 9029 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskUci3DU/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781966829091300-388-0/minicluster-data/ts-2/data,/tmp/dist-test-taskUci3DU/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781966829091300-388-0/minicluster-data/ts-2/wal
uuid: "5f650bd720844ca18d8ffda594914597"
format_stamp: "Formatted at 2026-06-20 14:48:21 on dist-test-slave-q9j6"
server_key: "4a16d24b8fa670f973b16484efa3e3d2"
server_key_iv: "423c0babded941b7e6fd7fad47566dff"
server_key_version: "encryptionkey@0"
I20260620 14:48:21.973608 9029 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskUci3DU/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781966829091300-388-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskUci3DU/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781966829091300-388-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskUci3DU/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781966829091300-388-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 14:48:21.989028 9029 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 14:48:21.993746 9029 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 14:48:21.994109 9029 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 14:48:21.995144 9029 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 14:48:21.996694 9029 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 14:48:21.996790 9029 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 14:48:21.996951 9029 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 14:48:21.997022 9029 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 14:48:22.029114 9029 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.3:34719
I20260620 14:48:22.029155 9156 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.3:34719 every 8 connection(s)
I20260620 14:48:22.030324 8889 heartbeater.cc:503] Master 127.0.97.62:41725 was elected leader, sending a full tablet report...
I20260620 14:48:22.030555 9029 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskUci3DU/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781966829091300-388-0/minicluster-data/ts-2/data/info.pb
I20260620 14:48:22.033746 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskUci3DU/build/asan/bin/kudu as pid 9029
I20260620 14:48:22.033948 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskUci3DU/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781966829091300-388-0/minicluster-data/ts-2/wal/instance
I20260620 14:48:22.034369 388 external_mini_cluster.cc:1428] Setting key 603cf861a58c5ad3599b4eaec589c9f8
Jun 20 14:48:22 dist-test-slave-q9j6 krb5kdc[8655](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781966901, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.3@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260620 14:48:22.051520 9025 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:48:22.033621 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.3:56451 (local address 127.0.97.62:41725)
0620 14:48:22.033847 (+ 226us) server_negotiation.cc:206] Beginning negotiation
0620 14:48:22.033854 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0620 14:48:22.035371 (+ 1517us) server_negotiation.cc:407] Connection header received
0620 14:48:22.037465 (+ 2094us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:48:22.037469 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:48:22.037535 (+ 66us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:48:22.037615 (+ 80us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:48:22.039324 (+ 1709us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:48:22.040119 (+ 795us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:48:22.041655 (+ 1536us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:48:22.042035 (+ 380us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:48:22.042124 (+ 89us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:48:22.045653 (+ 3529us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 14:48:22.045658 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 14:48:22.045661 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 14:48:22.045695 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0620 14:48:22.048411 (+ 2716us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:48:22.049177 (+ 766us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:48:22.049184 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:48:22.049188 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:48:22.049320 (+ 132us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:48:22.049781 (+ 461us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:48:22.049786 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:48:22.049790 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:48:22.050072 (+ 282us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 14:48:22.050248 (+ 176us) server_negotiation.cc:1035] Waiting for connection context
0620 14:48:22.050954 (+ 706us) server_negotiation.cc:299] Negotiation successful
0620 14:48:22.051181 (+ 227us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":69}
I20260620 14:48:22.052585 9159 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:48:22.034277 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:41725 (local address 127.0.97.3:56451)
0620 14:48:22.035206 (+ 929us) negotiation.cc:107] Waiting for socket to connect
0620 14:48:22.035255 (+ 49us) client_negotiation.cc:174] Beginning negotiation
0620 14:48:22.037168 (+ 1913us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 14:48:22.037854 (+ 686us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 14:48:22.037862 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 14:48:22.038564 (+ 702us) client_negotiation.cc:189] Negotiated authn=SASL
0620 14:48:22.039169 (+ 605us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:48:22.039187 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:48:22.040259 (+ 1072us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:48:22.040263 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:48:22.041528 (+ 1265us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:48:22.041540 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:48:22.042223 (+ 683us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:48:22.042227 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:48:22.042445 (+ 218us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:48:22.043108 (+ 663us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 14:48:22.043131 (+ 23us) client_negotiation.cc:656] calling sasl_client_start()
0620 14:48:22.045424 (+ 2293us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 14:48:22.048557 (+ 3133us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 14:48:22.048567 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 14:48:22.048594 (+ 27us) client_negotiation.cc:760] calling sasl_client_step()
0620 14:48:22.049044 (+ 450us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 14:48:22.049435 (+ 391us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 14:48:22.049440 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 14:48:22.049444 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0620 14:48:22.049650 (+ 206us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 14:48:22.050202 (+ 552us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 14:48:22.050211 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 14:48:22.050585 (+ 374us) client_negotiation.cc:769] Sending connection context
0620 14:48:22.050872 (+ 287us) client_negotiation.cc:240] Negotiation successful
0620 14:48:22.051434 (+ 562us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":634,"thread_start_us":396,"threads_started":1}
I20260620 14:48:22.054171 9157 heartbeater.cc:347] Connected to a master server at 127.0.97.62:41725
I20260620 14:48:22.054505 9157 heartbeater.cc:464] Registering TS with master...
I20260620 14:48:22.055274 9157 heartbeater.cc:511] Master 127.0.97.62:41725 requested a full tablet report, sending...
I20260620 14:48:22.056860 8700 ts_manager.cc:194] Registered new tserver with Master: 5f650bd720844ca18d8ffda594914597 (127.0.97.3:34719)
I20260620 14:48:22.057698 8700 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.97.3@KRBTEST.COM'} at 127.0.97.3:56451
I20260620 14:48:22.061995 388 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
WARNING: no policy specified for oryx/127.0.97.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.0.97.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.0.97.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskUci3DU/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781966829091300-388-0/krb5kdc/oryx_127.0.97.1.keytab.
Entry for principal oryx/127.0.97.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskUci3DU/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781966829091300-388-0/krb5kdc/oryx_127.0.97.1.keytab.
WARNING: no policy specified for oryx/127.0.97.2@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.0.97.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.0.97.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskUci3DU/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781966829091300-388-0/krb5kdc/oryx_127.0.97.2.keytab.
Entry for principal oryx/127.0.97.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskUci3DU/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781966829091300-388-0/krb5kdc/oryx_127.0.97.2.keytab.
WARNING: no policy specified for oryx/127.0.97.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.0.97.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.0.97.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskUci3DU/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781966829091300-388-0/krb5kdc/oryx_127.0.97.3.keytab.
Entry for principal oryx/127.0.97.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskUci3DU/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781966829091300-388-0/krb5kdc/oryx_127.0.97.3.keytab.
I20260620 14:48:22.200647 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskUci3DU/build/asan/bin/kudu pid 8761
Jun 20 14:48:22 dist-test-slave-q9j6 krb5kdc[8655](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781966899, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.1@KRBTEST.COM
I20260620 14:48:22.217051 9167 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:48:22.201907 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46528 (local address 127.0.97.1:33589)
0620 14:48:22.202542 (+ 635us) server_negotiation.cc:206] Beginning negotiation
0620 14:48:22.202549 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0620 14:48:22.202578 (+ 29us) server_negotiation.cc:407] Connection header received
0620 14:48:22.202640 (+ 62us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:48:22.202646 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:48:22.202912 (+ 266us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:48:22.203049 (+ 137us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:48:22.204335 (+ 1286us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:48:22.205341 (+ 1006us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:48:22.206576 (+ 1235us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:48:22.207134 (+ 558us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:48:22.207231 (+ 97us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:48:22.211284 (+ 4053us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 14:48:22.211297 (+ 13us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 14:48:22.211314 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 14:48:22.211357 (+ 43us) server_negotiation.cc:922] calling sasl_server_start()
0620 14:48:22.214495 (+ 3138us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:48:22.215193 (+ 698us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:48:22.215201 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:48:22.215203 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:48:22.215331 (+ 128us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:48:22.215725 (+ 394us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:48:22.215728 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:48:22.215729 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:48:22.216075 (+ 346us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 14:48:22.216294 (+ 219us) server_negotiation.cc:1035] Waiting for connection context
0620 14:48:22.216551 (+ 257us) server_negotiation.cc:299] Negotiation successful
0620 14:48:22.216748 (+ 197us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":486,"thread_start_us":349,"threads_started":1}
I20260620 14:48:22.218418 8884 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 14:48:22.301877 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskUci3DU/build/asan/bin/kudu with pid 8761
I20260620 14:48:22.318152 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskUci3DU/build/asan/bin/kudu pid 8895
Jun 20 14:48:22 dist-test-slave-q9j6 krb5kdc[8655](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781966899, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.2@KRBTEST.COM
I20260620 14:48:22.333494 9169 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:48:22.318985 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56640 (local address 127.0.97.2:40895)
0620 14:48:22.319559 (+ 574us) server_negotiation.cc:206] Beginning negotiation
0620 14:48:22.319566 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0620 14:48:22.319639 (+ 73us) server_negotiation.cc:407] Connection header received
0620 14:48:22.319719 (+ 80us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:48:22.319726 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:48:22.319945 (+ 219us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:48:22.320081 (+ 136us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:48:22.321270 (+ 1189us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:48:22.322273 (+ 1003us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:48:22.323518 (+ 1245us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:48:22.324180 (+ 662us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:48:22.324278 (+ 98us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:48:22.328251 (+ 3973us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 14:48:22.328263 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 14:48:22.328278 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 14:48:22.328311 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0620 14:48:22.331136 (+ 2825us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:48:22.331789 (+ 653us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:48:22.331796 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:48:22.331798 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:48:22.331916 (+ 118us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:48:22.332237 (+ 321us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:48:22.332240 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:48:22.332242 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:48:22.332554 (+ 312us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 14:48:22.332783 (+ 229us) server_negotiation.cc:1035] Waiting for connection context
0620 14:48:22.333026 (+ 243us) server_negotiation.cc:299] Negotiation successful
0620 14:48:22.333212 (+ 186us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":434,"thread_start_us":330,"threads_started":1}
I20260620 14:48:22.334755 9018 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 14:48:22.424494 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskUci3DU/build/asan/bin/kudu with pid 8895
I20260620 14:48:22.441016 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskUci3DU/build/asan/bin/kudu pid 9029
Jun 20 14:48:22 dist-test-slave-q9j6 krb5kdc[8655](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781966899, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.3@KRBTEST.COM
I20260620 14:48:22.458129 9171 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:48:22.442062 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33170 (local address 127.0.97.3:34719)
0620 14:48:22.442683 (+ 621us) server_negotiation.cc:206] Beginning negotiation
0620 14:48:22.442690 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0620 14:48:22.442743 (+ 53us) server_negotiation.cc:407] Connection header received
0620 14:48:22.442802 (+ 59us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:48:22.442808 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:48:22.443031 (+ 223us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:48:22.443230 (+ 199us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:48:22.444648 (+ 1418us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:48:22.445682 (+ 1034us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:48:22.447031 (+ 1349us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:48:22.447843 (+ 812us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:48:22.447997 (+ 154us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:48:22.452538 (+ 4541us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 14:48:22.452552 (+ 14us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 14:48:22.452568 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 14:48:22.452610 (+ 42us) server_negotiation.cc:922] calling sasl_server_start()
0620 14:48:22.455663 (+ 3053us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:48:22.456372 (+ 709us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:48:22.456379 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:48:22.456390 (+ 11us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:48:22.456519 (+ 129us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:48:22.456845 (+ 326us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:48:22.456848 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:48:22.456850 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:48:22.457189 (+ 339us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 14:48:22.457444 (+ 255us) server_negotiation.cc:1035] Waiting for connection context
0620 14:48:22.457636 (+ 192us) server_negotiation.cc:299] Negotiation successful
0620 14:48:22.457835 (+ 199us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":481,"thread_start_us":337,"threads_started":1}
I20260620 14:48:22.459493 9152 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 14:48:22.544075 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskUci3DU/build/asan/bin/kudu with pid 9029
I20260620 14:48:22.560220 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskUci3DU/build/asan/bin/kudu pid 8668
I20260620 14:48:22.561458 8730 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 14:48:22.667944 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskUci3DU/build/asan/bin/kudu with pid 8668
I20260620 14:48:22.688419 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskUci3DU/build/asan/bin/kudu
/tmp/dist-test-taskUci3DU/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskUci3DU/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781966829091300-388-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskUci3DU/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781966829091300-388-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskUci3DU/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781966829091300-388-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskUci3DU/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781966829091300-388-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:41725
--webserver_interface=127.0.97.62
--webserver_port=42213
--keytab_file=/tmp/dist-test-taskUci3DU/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781966829091300-388-0/krb5kdc/kudu_127.0.97.62.keytab
--principal=kudu/127.0.97.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:39945
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.97.62:41725
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskUci3DU/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781966829091300-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskUci3DU/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781966829091300-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskUci3DU/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781966829091300-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 14:48:23.019582 9174 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 14:48:23.019997 9174 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 14:48:23.020145 9174 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 14:48:23.027503 9174 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 14:48:23.027673 9174 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 14:48:23.027772 9174 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 14:48:23.027872 9174 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 14:48:23.027948 9174 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 14:48:23.038573 9174 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:39945
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskUci3DU/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781966829091300-388-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskUci3DU/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781966829091300-388-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.97.62:41725
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskUci3DU/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781966829091300-388-0/krb5kdc/kudu_127.0.97.62.keytab
--principal=kudu/127.0.97.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:41725
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskUci3DU/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781966829091300-388-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.97.62
--webserver_port=42213
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskUci3DU/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781966829091300-388-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 13:41:20 UTC on ccycloud.quasar-vdagoi.root.comops.site
build id 4457462
ASAN enabled
I20260620 14:48:23.040074 9174 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 14:48:23.041428 9174 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 14:48:23.053475 9180 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 14:48:23.053393 9182 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 14:48:23.053612 9179 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 14:48:23.054770 9174 server_base.cc:1034] running on GCE node
I20260620 14:48:23.055557 9174 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 14:48:23.057173 9174 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 14:48:23.058393 9174 hybrid_clock.cc:648] HybridClock initialized: now 1781966903058337 us; error 56 us; skew 500 ppm
I20260620 14:48:23.058718 9174 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 14:48:23 dist-test-slave-q9j6 krb5kdc[8655](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781966903, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 14:48:23.061909 9174 init.cc:377] Logged in from keytab as kudu/127.0.97.62@KRBTEST.COM (short username kudu)
I20260620 14:48:23.063462 9174 webserver.cc:460] Webserver started at http://127.0.97.62:42213/ using document root <none> and password file <none>
I20260620 14:48:23.064260 9174 fs_manager.cc:359] Metadata directory not provided
I20260620 14:48:23.064427 9174 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 14:48:23.070178 9174 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.005s sys 0.000s
I20260620 14:48:23.073096 9189 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 14:48:23.074461 9174 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260620 14:48:23.074651 9174 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskUci3DU/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781966829091300-388-0/minicluster-data/master-0/data,/tmp/dist-test-taskUci3DU/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781966829091300-388-0/minicluster-data/master-0/wal
uuid: "2538e6ad3f224be8952920801072ebb3"
format_stamp: "Formatted at 2026-06-20 14:48:20 on dist-test-slave-q9j6"
server_key: "0b3ccd5bc5ffcc7e0ea503d3052c6903"
server_key_iv: "5ad5aa4993a98fdddab83bd19532e05b"
server_key_version: "encryptionkey@0"
I20260620 14:48:23.075671 9174 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskUci3DU/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781966829091300-388-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskUci3DU/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781966829091300-388-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskUci3DU/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781966829091300-388-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 14:48:23.110623 9174 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 14:48:23.114831 9174 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 14:48:23.115131 9174 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 14:48:23.134559 9174 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.62:41725
I20260620 14:48:23.134567 9240 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.62:41725 every 8 connection(s)
I20260620 14:48:23.136210 9174 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskUci3DU/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781966829091300-388-0/minicluster-data/master-0/data/info.pb
I20260620 14:48:23.141274 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskUci3DU/build/asan/bin/kudu as pid 9174
I20260620 14:48:23.142057 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskUci3DU/build/asan/bin/kudu
/tmp/dist-test-taskUci3DU/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskUci3DU/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781966829091300-388-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskUci3DU/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781966829091300-388-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskUci3DU/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781966829091300-388-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskUci3DU/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781966829091300-388-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.1:33589
--local_ip_for_outbound_sockets=127.0.97.1
--tserver_master_addrs=127.0.97.62:41725
--webserver_port=36279
--webserver_interface=127.0.97.1
--keytab_file=/tmp/dist-test-taskUci3DU/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781966829091300-388-0/krb5kdc/kudu_127.0.97.1.keytab
--principal=kudu/127.0.97.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:39945
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--principal=oryx/127.0.97.1
--keytab_file=/tmp/dist-test-taskUci3DU/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781966829091300-388-0/krb5kdc/oryx_127.0.97.1.keytab with env {KRB5CCNAME=/tmp/dist-test-taskUci3DU/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781966829091300-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskUci3DU/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781966829091300-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskUci3DU/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781966829091300-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260620 14:48:23.143769 9241 sys_catalog.cc:263] Verifying existing consensus state
I20260620 14:48:23.148499 9241 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 2538e6ad3f224be8952920801072ebb3: Bootstrap starting.
I20260620 14:48:23.175989 9241 log.cc:824] T 00000000000000000000000000000000 P 2538e6ad3f224be8952920801072ebb3: Log is configured to *not* fsync() on all Append() calls
I20260620 14:48:23.187672 9241 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 2538e6ad3f224be8952920801072ebb3: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=3 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260620 14:48:23.188320 9241 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 2538e6ad3f224be8952920801072ebb3: Bootstrap complete.
I20260620 14:48:23.198331 9241 raft_consensus.cc:348] T 00000000000000000000000000000000 P 2538e6ad3f224be8952920801072ebb3 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2538e6ad3f224be8952920801072ebb3" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 41725 } }
I20260620 14:48:23.199189 9241 raft_consensus.cc:729] T 00000000000000000000000000000000 P 2538e6ad3f224be8952920801072ebb3 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2538e6ad3f224be8952920801072ebb3, State: Initialized, Role: FOLLOWER
I20260620 14:48:23.199908 9241 consensus_queue.cc:260] T 00000000000000000000000000000000 P 2538e6ad3f224be8952920801072ebb3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2538e6ad3f224be8952920801072ebb3" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 41725 } }
I20260620 14:48:23.200171 9241 raft_consensus.cc:388] T 00000000000000000000000000000000 P 2538e6ad3f224be8952920801072ebb3 [term 1 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 14:48:23.200306 9241 raft_consensus.cc:482] T 00000000000000000000000000000000 P 2538e6ad3f224be8952920801072ebb3 [term 1 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 14:48:23.200488 9241 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 2538e6ad3f224be8952920801072ebb3 [term 1 FOLLOWER]: Advancing to term 2
I20260620 14:48:23.204589 9241 raft_consensus.cc:504] T 00000000000000000000000000000000 P 2538e6ad3f224be8952920801072ebb3 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2538e6ad3f224be8952920801072ebb3" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 41725 } }
I20260620 14:48:23.205156 9241 leader_election.cc:302] T 00000000000000000000000000000000 P 2538e6ad3f224be8952920801072ebb3 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 2538e6ad3f224be8952920801072ebb3; no voters:
I20260620 14:48:23.205825 9241 leader_election.cc:288] T 00000000000000000000000000000000 P 2538e6ad3f224be8952920801072ebb3 [CANDIDATE]: Term 2 election: Requested vote from peers
I20260620 14:48:23.205973 9245 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 2538e6ad3f224be8952920801072ebb3 [term 2 FOLLOWER]: Leader election won for term 2
I20260620 14:48:23.207255 9245 raft_consensus.cc:686] T 00000000000000000000000000000000 P 2538e6ad3f224be8952920801072ebb3 [term 2 LEADER]: Becoming Leader. State: Replica: 2538e6ad3f224be8952920801072ebb3, State: Running, Role: LEADER
I20260620 14:48:23.207924 9245 consensus_queue.cc:237] T 00000000000000000000000000000000 P 2538e6ad3f224be8952920801072ebb3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 2, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2538e6ad3f224be8952920801072ebb3" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 41725 } }
I20260620 14:48:23.208232 9241 sys_catalog.cc:564] T 00000000000000000000000000000000 P 2538e6ad3f224be8952920801072ebb3 [sys.catalog]: configured and running, proceeding with master startup.
I20260620 14:48:23.210387 9247 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2538e6ad3f224be8952920801072ebb3 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 2538e6ad3f224be8952920801072ebb3. Latest consensus state: current_term: 2 leader_uuid: "2538e6ad3f224be8952920801072ebb3" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2538e6ad3f224be8952920801072ebb3" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 41725 } } }
I20260620 14:48:23.210752 9247 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2538e6ad3f224be8952920801072ebb3 [sys.catalog]: This master's current role is: LEADER
I20260620 14:48:23.211400 9246 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2538e6ad3f224be8952920801072ebb3 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 2 leader_uuid: "2538e6ad3f224be8952920801072ebb3" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2538e6ad3f224be8952920801072ebb3" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 41725 } } }
I20260620 14:48:23.211716 9246 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2538e6ad3f224be8952920801072ebb3 [sys.catalog]: This master's current role is: LEADER
I20260620 14:48:23.217166 9252 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 14:48:23.224017 9252 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 14:48:23.226017 9252 catalog_manager.cc:1259] Loaded cluster ID: efc1fcfe5d0c408684a4e9a7af42f424
I20260620 14:48:23.226261 9252 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 14:48:23.231626 9252 catalog_manager.cc:1504] Loading token signing keys...
I20260620 14:48:23.233821 9252 catalog_manager.cc:5879] T 00000000000000000000000000000000 P 2538e6ad3f224be8952920801072ebb3: Loaded TSK: 0
I20260620 14:48:23.235919 9252 catalog_manager.cc:1514] Initializing in-progress tserver states...
W20260620 14:48:23.522692 9243 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 14:48:23.523067 9243 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 14:48:23.523252 9243 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 14:48:23.530784 9243 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 14:48:23.530958 9243 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 14:48:23.531152 9243 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.1
I20260620 14:48:23.541273 9243 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:39945
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskUci3DU/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781966829091300-388-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskUci3DU/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781966829091300-388-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskUci3DU/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781966829091300-388-0/krb5kdc/oryx_127.0.97.1.keytab
--principal=oryx/127.0.97.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.1:33589
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskUci3DU/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781966829091300-388-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.97.1
--webserver_port=36279
--tserver_master_addrs=127.0.97.62:41725
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.1
--log_dir=/tmp/dist-test-taskUci3DU/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781966829091300-388-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 13:41:20 UTC on ccycloud.quasar-vdagoi.root.comops.site
build id 4457462
ASAN enabled
I20260620 14:48:23.542681 9243 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 14:48:23.544104 9243 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 14:48:23.559259 9267 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 14:48:23.559435 9268 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 14:48:23.559715 9270 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 14:48:23.560757 9243 server_base.cc:1034] running on GCE node
I20260620 14:48:23.561473 9243 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 14:48:23.562467 9243 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 14:48:23.563711 9243 hybrid_clock.cc:648] HybridClock initialized: now 1781966903563638 us; error 59 us; skew 500 ppm
I20260620 14:48:23.564059 9243 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 14:48:23 dist-test-slave-q9j6 krb5kdc[8655](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781966903, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.97.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 14:48:23.567442 9243 init.cc:377] Logged in from keytab as oryx/127.0.97.1@KRBTEST.COM (short username oryx)
I20260620 14:48:23.569168 9243 webserver.cc:460] Webserver started at http://127.0.97.1:36279/ using document root <none> and password file <none>
I20260620 14:48:23.569904 9243 fs_manager.cc:359] Metadata directory not provided
I20260620 14:48:23.570060 9243 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 14:48:23.575579 9243 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.003s sys 0.000s
I20260620 14:48:23.578518 9277 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 14:48:23.580173 9243 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.002s
I20260620 14:48:23.580356 9243 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskUci3DU/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781966829091300-388-0/minicluster-data/ts-0/data,/tmp/dist-test-taskUci3DU/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781966829091300-388-0/minicluster-data/ts-0/wal
uuid: "3e929367ca2a4cbd8f573833779da5a5"
format_stamp: "Formatted at 2026-06-20 14:48:20 on dist-test-slave-q9j6"
server_key: "25bf6cf522b1578294962344ecb80e47"
server_key_iv: "36be690accabd96a85693fe246465faa"
server_key_version: "encryptionkey@0"
I20260620 14:48:23.581317 9243 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskUci3DU/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781966829091300-388-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskUci3DU/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781966829091300-388-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskUci3DU/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781966829091300-388-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 14:48:23.613929 9243 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 14:48:23.618069 9243 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 14:48:23.618362 9243 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 14:48:23.619381 9243 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 14:48:23.620863 9243 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 14:48:23.620988 9243 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 14:48:23.621121 9243 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 14:48:23.621222 9243 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 14:48:23.654700 9243 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.1:33589
I20260620 14:48:23.654758 9389 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.1:33589 every 8 connection(s)
I20260620 14:48:23.656299 9243 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskUci3DU/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781966829091300-388-0/minicluster-data/ts-0/data/info.pb
I20260620 14:48:23.660894 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskUci3DU/build/asan/bin/kudu as pid 9243
I20260620 14:48:23.661477 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskUci3DU/build/asan/bin/kudu
/tmp/dist-test-taskUci3DU/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskUci3DU/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781966829091300-388-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskUci3DU/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781966829091300-388-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskUci3DU/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781966829091300-388-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskUci3DU/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781966829091300-388-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.2:40895
--local_ip_for_outbound_sockets=127.0.97.2
--tserver_master_addrs=127.0.97.62:41725
--webserver_port=41337
--webserver_interface=127.0.97.2
--keytab_file=/tmp/dist-test-taskUci3DU/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781966829091300-388-0/krb5kdc/kudu_127.0.97.2.keytab
--principal=kudu/127.0.97.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:39945
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--principal=oryx/127.0.97.2
--keytab_file=/tmp/dist-test-taskUci3DU/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781966829091300-388-0/krb5kdc/oryx_127.0.97.2.keytab with env {KRB5CCNAME=/tmp/dist-test-taskUci3DU/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781966829091300-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskUci3DU/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781966829091300-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskUci3DU/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781966829091300-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Jun 20 14:48:23 dist-test-slave-q9j6 krb5kdc[8655](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.1@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
Jun 20 14:48:23 dist-test-slave-q9j6 krb5kdc[8655](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.1@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
I20260620 14:48:23.675136 9393 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:48:23.659506 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:41725 (local address 127.0.97.1:43311)
0620 14:48:23.660380 (+ 874us) negotiation.cc:107] Waiting for socket to connect
0620 14:48:23.660427 (+ 47us) client_negotiation.cc:174] Beginning negotiation
0620 14:48:23.662269 (+ 1842us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 14:48:23.663781 (+ 1512us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 14:48:23.663793 (+ 12us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 14:48:23.664412 (+ 619us) client_negotiation.cc:189] Negotiated authn=SASL
0620 14:48:23.664979 (+ 567us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:48:23.664998 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:48:23.666675 (+ 1677us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:48:23.666680 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:48:23.667998 (+ 1318us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:48:23.668011 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:48:23.669080 (+ 1069us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:48:23.669088 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:48:23.669370 (+ 282us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:48:23.670094 (+ 724us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 14:48:23.670117 (+ 23us) client_negotiation.cc:656] calling sasl_client_start()
0620 14:48:23.673770 (+ 3653us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.97.62:41725: Server oryx/127.0.97.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":560,"thread_start_us":362,"threads_started":1}
W20260620 14:48:23.675163 9392 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 14:48:23.659386 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.1:43311 (local address 127.0.97.62:41725)
0620 14:48:23.660255 (+ 869us) server_negotiation.cc:206] Beginning negotiation
0620 14:48:23.660297 (+ 42us) server_negotiation.cc:399] Waiting for connection header
0620 14:48:23.660562 (+ 265us) server_negotiation.cc:407] Connection header received
0620 14:48:23.662850 (+ 2288us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:48:23.662864 (+ 14us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:48:23.663220 (+ 356us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:48:23.663601 (+ 381us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:48:23.665207 (+ 1606us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:48:23.666473 (+ 1266us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:48:23.668155 (+ 1682us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:48:23.668820 (+ 665us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:48:23.669047 (+ 227us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:48:23.674168 (+ 5121us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.97.1:43311: BlockingRecv error: recv got EOF from 127.0.97.1:43311 (error 108)
Metrics: {"server-negotiator.queue_time_us":601,"thread_start_us":349,"threads_started":1}
W20260620 14:48:23.675977 9390 heartbeater.cc:644] Failed to heartbeat to 127.0.97.62:41725 (0 consecutive failures): Not authorized: Failed to ping master at 127.0.97.62:41725: Client connection negotiation failed: client connection to 127.0.97.62:41725: Server oryx/127.0.97.62@KRBTEST.COM not found in Kerberos database
Jun 20 14:48:23 dist-test-slave-q9j6 krb5kdc[8655](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.1@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
Jun 20 14:48:23 dist-test-slave-q9j6 krb5kdc[8655](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.1@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
I20260620 14:48:23.684610 9393 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:48:23.676650 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:41725 (local address 127.0.97.1:51327)
0620 14:48:23.676800 (+ 150us) negotiation.cc:107] Waiting for socket to connect
0620 14:48:23.676819 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0620 14:48:23.676946 (+ 127us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 14:48:23.677302 (+ 356us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 14:48:23.677306 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 14:48:23.677717 (+ 411us) client_negotiation.cc:189] Negotiated authn=SASL
0620 14:48:23.677860 (+ 143us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:48:23.677868 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:48:23.678935 (+ 1067us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:48:23.678940 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:48:23.679988 (+ 1048us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:48:23.680000 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:48:23.680700 (+ 700us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:48:23.680706 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:48:23.680889 (+ 183us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:48:23.681522 (+ 633us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 14:48:23.681539 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0620 14:48:23.684306 (+ 2767us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.97.62:41725: Server oryx/127.0.97.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":46}
W20260620 14:48:23.684729 9392 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 14:48:23.676858 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.1:51327 (local address 127.0.97.62:41725)
0620 14:48:23.677053 (+ 195us) server_negotiation.cc:206] Beginning negotiation
0620 14:48:23.677058 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 14:48:23.677077 (+ 19us) server_negotiation.cc:407] Connection header received
0620 14:48:23.677121 (+ 44us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:48:23.677124 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:48:23.677175 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:48:23.677274 (+ 99us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:48:23.678010 (+ 736us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:48:23.678785 (+ 775us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:48:23.680135 (+ 1350us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:48:23.680553 (+ 418us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:48:23.680687 (+ 134us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:48:23.684516 (+ 3829us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.97.1:51327: BlockingRecv error: recv got EOF from 127.0.97.1:51327 (error 108)
Metrics: {"server-negotiator.queue_time_us":66}
Jun 20 14:48:23 dist-test-slave-q9j6 krb5kdc[8655](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.1@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
Jun 20 14:48:23 dist-test-slave-q9j6 krb5kdc[8655](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.1@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
I20260620 14:48:23.693676 9393 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:48:23.685436 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:41725 (local address 127.0.97.1:50761)
0620 14:48:23.685600 (+ 164us) negotiation.cc:107] Waiting for socket to connect
0620 14:48:23.685618 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0620 14:48:23.685706 (+ 88us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 14:48:23.686188 (+ 482us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 14:48:23.686192 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 14:48:23.686458 (+ 266us) client_negotiation.cc:189] Negotiated authn=SASL
0620 14:48:23.686600 (+ 142us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:48:23.686608 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:48:23.688081 (+ 1473us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:48:23.688084 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:48:23.689015 (+ 931us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:48:23.689028 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:48:23.689939 (+ 911us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:48:23.689944 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:48:23.690066 (+ 122us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:48:23.690666 (+ 600us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 14:48:23.690683 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0620 14:48:23.693457 (+ 2774us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.97.62:41725: Server oryx/127.0.97.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":50}
W20260620 14:48:23.694006 9392 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 14:48:23.685500 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.1:50761 (local address 127.0.97.62:41725)
0620 14:48:23.685850 (+ 350us) server_negotiation.cc:206] Beginning negotiation
0620 14:48:23.685857 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0620 14:48:23.685876 (+ 19us) server_negotiation.cc:407] Connection header received
0620 14:48:23.685958 (+ 82us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:48:23.685963 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:48:23.686044 (+ 81us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:48:23.686152 (+ 108us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:48:23.686758 (+ 606us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:48:23.687939 (+ 1181us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:48:23.689170 (+ 1231us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:48:23.689796 (+ 626us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:48:23.689930 (+ 134us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:48:23.693720 (+ 3790us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.97.1:50761: BlockingRecv error: recv got EOF from 127.0.97.1:50761 (error 108)
Metrics: {"server-negotiator.queue_time_us":243}
W20260620 14:48:23.694134 9390 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260620 14:48:24.000101 9394 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 14:48:24.000379 9394 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 14:48:24.000567 9394 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 14:48:24.008003 9394 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 14:48:24.008150 9394 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 14:48:24.008348 9394 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.2
I20260620 14:48:24.018165 9394 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:39945
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskUci3DU/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781966829091300-388-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskUci3DU/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781966829091300-388-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskUci3DU/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781966829091300-388-0/krb5kdc/oryx_127.0.97.2.keytab
--principal=oryx/127.0.97.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.2:40895
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskUci3DU/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781966829091300-388-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.97.2
--webserver_port=41337
--tserver_master_addrs=127.0.97.62:41725
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.2
--log_dir=/tmp/dist-test-taskUci3DU/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781966829091300-388-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 13:41:20 UTC on ccycloud.quasar-vdagoi.root.comops.site
build id 4457462
ASAN enabled
I20260620 14:48:24.019444 9394 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 14:48:24.020687 9394 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 14:48:24.033799 9399 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 14:48:24.033804 9402 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 14:48:24.033812 9400 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 14:48:24.034674 9394 server_base.cc:1034] running on GCE node
I20260620 14:48:24.035405 9394 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 14:48:24.036450 9394 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 14:48:24.037663 9394 hybrid_clock.cc:648] HybridClock initialized: now 1781966904037589 us; error 63 us; skew 500 ppm
I20260620 14:48:24.038008 9394 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 14:48:24 dist-test-slave-q9j6 krb5kdc[8655](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781966904, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.97.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 14:48:24.041257 9394 init.cc:377] Logged in from keytab as oryx/127.0.97.2@KRBTEST.COM (short username oryx)
I20260620 14:48:24.042837 9394 webserver.cc:460] Webserver started at http://127.0.97.2:41337/ using document root <none> and password file <none>
I20260620 14:48:24.043695 9394 fs_manager.cc:359] Metadata directory not provided
I20260620 14:48:24.043857 9394 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 14:48:24.049612 9394 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.004s sys 0.000s
I20260620 14:48:24.052594 9409 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 14:48:24.054025 9394 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.001s
I20260620 14:48:24.054222 9394 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskUci3DU/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781966829091300-388-0/minicluster-data/ts-1/data,/tmp/dist-test-taskUci3DU/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781966829091300-388-0/minicluster-data/ts-1/wal
uuid: "4d756e74cf814ef184391921f8c2bb54"
format_stamp: "Formatted at 2026-06-20 14:48:21 on dist-test-slave-q9j6"
server_key: "080abc3796168454037b682c3493a12d"
server_key_iv: "d8a19159c631ff0a89703c12235b190a"
server_key_version: "encryptionkey@0"
I20260620 14:48:24.055226 9394 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskUci3DU/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781966829091300-388-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskUci3DU/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781966829091300-388-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskUci3DU/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781966829091300-388-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 14:48:24.073067 9394 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 14:48:24.077286 9394 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 14:48:24.077584 9394 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 14:48:24.078574 9394 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 14:48:24.080113 9394 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 14:48:24.080240 9394 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 14:48:24.080379 9394 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 14:48:24.080478 9394 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 14:48:24.114950 9394 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.2:40895
I20260620 14:48:24.115013 9521 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.2:40895 every 8 connection(s)
I20260620 14:48:24.116564 9394 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskUci3DU/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781966829091300-388-0/minicluster-data/ts-1/data/info.pb
I20260620 14:48:24.124959 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskUci3DU/build/asan/bin/kudu as pid 9394
I20260620 14:48:24.125538 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskUci3DU/build/asan/bin/kudu
/tmp/dist-test-taskUci3DU/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskUci3DU/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781966829091300-388-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskUci3DU/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781966829091300-388-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskUci3DU/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781966829091300-388-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskUci3DU/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781966829091300-388-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.3:34719
--local_ip_for_outbound_sockets=127.0.97.3
--tserver_master_addrs=127.0.97.62:41725
--webserver_port=39895
--webserver_interface=127.0.97.3
--keytab_file=/tmp/dist-test-taskUci3DU/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781966829091300-388-0/krb5kdc/kudu_127.0.97.3.keytab
--principal=kudu/127.0.97.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:39945
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--principal=oryx/127.0.97.3
--keytab_file=/tmp/dist-test-taskUci3DU/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781966829091300-388-0/krb5kdc/oryx_127.0.97.3.keytab with env {KRB5CCNAME=/tmp/dist-test-taskUci3DU/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781966829091300-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskUci3DU/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781966829091300-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskUci3DU/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781966829091300-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Jun 20 14:48:24 dist-test-slave-q9j6 krb5kdc[8655](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.2@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
Jun 20 14:48:24 dist-test-slave-q9j6 krb5kdc[8655](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.2@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
W20260620 14:48:24.134477 9392 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 14:48:24.119264 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.2:40503 (local address 127.0.97.62:41725)
0620 14:48:24.119484 (+ 220us) server_negotiation.cc:206] Beginning negotiation
0620 14:48:24.119490 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 14:48:24.120755 (+ 1265us) server_negotiation.cc:407] Connection header received
0620 14:48:24.122654 (+ 1899us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:48:24.122658 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:48:24.122738 (+ 80us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:48:24.122845 (+ 107us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:48:24.124767 (+ 1922us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:48:24.125547 (+ 780us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:48:24.128318 (+ 2771us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:48:24.128757 (+ 439us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:48:24.128861 (+ 104us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:48:24.134199 (+ 5338us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.97.2:40503: BlockingRecv error: recv got EOF from 127.0.97.2:40503 (error 108)
Metrics: {"server-negotiator.queue_time_us":71}
I20260620 14:48:24.135385 9524 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:48:24.119848 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:41725 (local address 127.0.97.2:40503)
0620 14:48:24.120584 (+ 736us) negotiation.cc:107] Waiting for socket to connect
0620 14:48:24.120622 (+ 38us) client_negotiation.cc:174] Beginning negotiation
0620 14:48:24.122328 (+ 1706us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 14:48:24.123151 (+ 823us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 14:48:24.123160 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 14:48:24.123819 (+ 659us) client_negotiation.cc:189] Negotiated authn=SASL
0620 14:48:24.124543 (+ 724us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:48:24.124572 (+ 29us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:48:24.125726 (+ 1154us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:48:24.125734 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:48:24.128117 (+ 2383us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:48:24.128139 (+ 22us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:48:24.129022 (+ 883us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:48:24.129033 (+ 11us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:48:24.129408 (+ 375us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:48:24.130241 (+ 833us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 14:48:24.130266 (+ 25us) client_negotiation.cc:656] calling sasl_client_start()
0620 14:48:24.133995 (+ 3729us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.97.62:41725: Server oryx/127.0.97.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":464,"thread_start_us":283,"threads_started":1}
W20260620 14:48:24.136143 9522 heartbeater.cc:644] Failed to heartbeat to 127.0.97.62:41725 (0 consecutive failures): Not authorized: Failed to ping master at 127.0.97.62:41725: Client connection negotiation failed: client connection to 127.0.97.62:41725: Server oryx/127.0.97.62@KRBTEST.COM not found in Kerberos database
Jun 20 14:48:24 dist-test-slave-q9j6 krb5kdc[8655](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.2@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
Jun 20 14:48:24 dist-test-slave-q9j6 krb5kdc[8655](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.2@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
I20260620 14:48:24.144740 9524 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:48:24.136851 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:41725 (local address 127.0.97.2:45349)
0620 14:48:24.137017 (+ 166us) negotiation.cc:107] Waiting for socket to connect
0620 14:48:24.137036 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0620 14:48:24.137139 (+ 103us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 14:48:24.137488 (+ 349us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 14:48:24.137491 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 14:48:24.137796 (+ 305us) client_negotiation.cc:189] Negotiated authn=SASL
0620 14:48:24.137941 (+ 145us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:48:24.137949 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:48:24.139010 (+ 1061us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:48:24.139015 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:48:24.139999 (+ 984us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:48:24.140011 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:48:24.140732 (+ 721us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:48:24.140738 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:48:24.140911 (+ 173us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:48:24.141545 (+ 634us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 14:48:24.141563 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0620 14:48:24.144443 (+ 2880us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.97.62:41725: Server oryx/127.0.97.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":53}
W20260620 14:48:24.144896 9392 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 14:48:24.136943 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.2:45349 (local address 127.0.97.62:41725)
0620 14:48:24.137140 (+ 197us) server_negotiation.cc:206] Beginning negotiation
0620 14:48:24.137148 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0620 14:48:24.137172 (+ 24us) server_negotiation.cc:407] Connection header received
0620 14:48:24.137236 (+ 64us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:48:24.137243 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:48:24.137330 (+ 87us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:48:24.137445 (+ 115us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:48:24.138092 (+ 647us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:48:24.138859 (+ 767us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:48:24.140152 (+ 1293us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:48:24.140578 (+ 426us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:48:24.140717 (+ 139us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:48:24.144648 (+ 3931us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.97.2:45349: BlockingRecv error: recv got EOF from 127.0.97.2:45349 (error 108)
Metrics: {"server-negotiator.queue_time_us":67}
Jun 20 14:48:24 dist-test-slave-q9j6 krb5kdc[8655](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.2@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
Jun 20 14:48:24 dist-test-slave-q9j6 krb5kdc[8655](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.2@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
I20260620 14:48:24.155483 9524 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:48:24.145681 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:41725 (local address 127.0.97.2:38351)
0620 14:48:24.145915 (+ 234us) negotiation.cc:107] Waiting for socket to connect
0620 14:48:24.145945 (+ 30us) client_negotiation.cc:174] Beginning negotiation
0620 14:48:24.146051 (+ 106us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 14:48:24.146375 (+ 324us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 14:48:24.146379 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 14:48:24.146693 (+ 314us) client_negotiation.cc:189] Negotiated authn=SASL
0620 14:48:24.146837 (+ 144us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:48:24.146845 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:48:24.148018 (+ 1173us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:48:24.148025 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:48:24.149395 (+ 1370us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:48:24.149414 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:48:24.150604 (+ 1190us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:48:24.150609 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:48:24.150812 (+ 203us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:48:24.151498 (+ 686us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 14:48:24.151526 (+ 28us) client_negotiation.cc:656] calling sasl_client_start()
0620 14:48:24.155154 (+ 3628us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.97.62:41725: Server oryx/127.0.97.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":128}
W20260620 14:48:24.155849 9522 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260620 14:48:24.155989 9392 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 14:48:24.145762 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.2:38351 (local address 127.0.97.62:41725)
0620 14:48:24.145917 (+ 155us) server_negotiation.cc:206] Beginning negotiation
0620 14:48:24.145924 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0620 14:48:24.145993 (+ 69us) server_negotiation.cc:407] Connection header received
0620 14:48:24.146183 (+ 190us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:48:24.146188 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:48:24.146249 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:48:24.146350 (+ 101us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:48:24.146984 (+ 634us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:48:24.147845 (+ 861us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:48:24.149583 (+ 1738us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:48:24.150208 (+ 625us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:48:24.150314 (+ 106us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:48:24.155375 (+ 5061us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.97.2:38351: BlockingRecv error: recv got EOF from 127.0.97.2:38351 (error 108)
Metrics: {"server-negotiator.queue_time_us":51}
W20260620 14:48:24.468540 9525 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 14:48:24.468900 9525 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 14:48:24.469069 9525 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 14:48:24.476460 9525 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 14:48:24.476630 9525 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 14:48:24.476821 9525 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.3
I20260620 14:48:24.486959 9525 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:39945
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskUci3DU/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781966829091300-388-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskUci3DU/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781966829091300-388-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskUci3DU/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781966829091300-388-0/krb5kdc/oryx_127.0.97.3.keytab
--principal=oryx/127.0.97.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.3:34719
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskUci3DU/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781966829091300-388-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.97.3
--webserver_port=39895
--tserver_master_addrs=127.0.97.62:41725
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.3
--log_dir=/tmp/dist-test-taskUci3DU/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781966829091300-388-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 13:41:20 UTC on ccycloud.quasar-vdagoi.root.comops.site
build id 4457462
ASAN enabled
I20260620 14:48:24.488395 9525 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 14:48:24.489742 9525 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 14:48:24.503093 9530 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 14:48:24.503067 9533 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 14:48:24.503047 9531 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 14:48:24.504248 9525 server_base.cc:1034] running on GCE node
I20260620 14:48:24.504855 9525 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 14:48:24.505820 9525 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 14:48:24.507045 9525 hybrid_clock.cc:648] HybridClock initialized: now 1781966904506998 us; error 51 us; skew 500 ppm
I20260620 14:48:24.507486 9525 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 14:48:24 dist-test-slave-q9j6 krb5kdc[8655](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781966904, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.97.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 14:48:24.510641 9525 init.cc:377] Logged in from keytab as oryx/127.0.97.3@KRBTEST.COM (short username oryx)
I20260620 14:48:24.512208 9525 webserver.cc:460] Webserver started at http://127.0.97.3:39895/ using document root <none> and password file <none>
I20260620 14:48:24.512984 9525 fs_manager.cc:359] Metadata directory not provided
I20260620 14:48:24.513139 9525 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 14:48:24.518962 9525 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.005s sys 0.000s
I20260620 14:48:24.521795 9540 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 14:48:24.523264 9525 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260620 14:48:24.523463 9525 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskUci3DU/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781966829091300-388-0/minicluster-data/ts-2/data,/tmp/dist-test-taskUci3DU/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781966829091300-388-0/minicluster-data/ts-2/wal
uuid: "5f650bd720844ca18d8ffda594914597"
format_stamp: "Formatted at 2026-06-20 14:48:21 on dist-test-slave-q9j6"
server_key: "4a16d24b8fa670f973b16484efa3e3d2"
server_key_iv: "423c0babded941b7e6fd7fad47566dff"
server_key_version: "encryptionkey@0"
I20260620 14:48:24.524574 9525 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskUci3DU/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781966829091300-388-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskUci3DU/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781966829091300-388-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskUci3DU/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781966829091300-388-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 14:48:24.539235 9525 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 14:48:24.543910 9525 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 14:48:24.544206 9525 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 14:48:24.545212 9525 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 14:48:24.546582 9525 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 14:48:24.546717 9525 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 14:48:24.546857 9525 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 14:48:24.546949 9525 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 14:48:24.580368 9525 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.3:34719
I20260620 14:48:24.580422 9652 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.3:34719 every 8 connection(s)
I20260620 14:48:24.582022 9525 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskUci3DU/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781966829091300-388-0/minicluster-data/ts-2/data/info.pb
I20260620 14:48:24.583091 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskUci3DU/build/asan/bin/kudu as pid 9525
Jun 20 14:48:24 dist-test-slave-q9j6 krb5kdc[8655](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.3@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
Jun 20 14:48:24 dist-test-slave-q9j6 krb5kdc[8655](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.3@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
W20260620 14:48:24.600263 9656 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 14:48:24.584966 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.3:35659 (local address 127.0.97.62:41725)
0620 14:48:24.585481 (+ 515us) server_negotiation.cc:206] Beginning negotiation
0620 14:48:24.585489 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0620 14:48:24.586475 (+ 986us) server_negotiation.cc:407] Connection header received
0620 14:48:24.589247 (+ 2772us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:48:24.589250 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:48:24.589300 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:48:24.589384 (+ 84us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:48:24.591295 (+ 1911us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:48:24.592098 (+ 803us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:48:24.593918 (+ 1820us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:48:24.594318 (+ 400us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:48:24.594456 (+ 138us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:48:24.600033 (+ 5577us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.97.3:35659: BlockingRecv error: recv got EOF from 127.0.97.3:35659 (error 108)
Metrics: {"server-negotiator.queue_time_us":368,"thread_start_us":267,"threads_started":1}
I20260620 14:48:24.601168 9657 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:48:24.585477 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:41725 (local address 127.0.97.3:35659)
0620 14:48:24.586290 (+ 813us) negotiation.cc:107] Waiting for socket to connect
0620 14:48:24.586337 (+ 47us) client_negotiation.cc:174] Beginning negotiation
0620 14:48:24.588915 (+ 2578us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 14:48:24.589732 (+ 817us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 14:48:24.589742 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 14:48:24.590264 (+ 522us) client_negotiation.cc:189] Negotiated authn=SASL
0620 14:48:24.591109 (+ 845us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:48:24.591137 (+ 28us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:48:24.592270 (+ 1133us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:48:24.592275 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:48:24.593761 (+ 1486us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:48:24.593778 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:48:24.594561 (+ 783us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:48:24.594568 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:48:24.594803 (+ 235us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:48:24.595467 (+ 664us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 14:48:24.595488 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0620 14:48:24.599762 (+ 4274us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.97.62:41725: Server oryx/127.0.97.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":524,"thread_start_us":368,"threads_started":1}
W20260620 14:48:24.601868 9653 heartbeater.cc:644] Failed to heartbeat to 127.0.97.62:41725 (0 consecutive failures): Not authorized: Failed to ping master at 127.0.97.62:41725: Client connection negotiation failed: client connection to 127.0.97.62:41725: Server oryx/127.0.97.62@KRBTEST.COM not found in Kerberos database
I20260620 14:48:24.602779 9392 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:48:24.584204 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58928 (local address 127.0.97.62:41725)
0620 14:48:24.584456 (+ 252us) server_negotiation.cc:206] Beginning negotiation
0620 14:48:24.584465 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0620 14:48:24.587297 (+ 2832us) server_negotiation.cc:407] Connection header received
0620 14:48:24.587477 (+ 180us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:48:24.587481 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:48:24.587556 (+ 75us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:48:24.587708 (+ 152us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:48:24.589657 (+ 1949us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:48:24.590329 (+ 672us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:48:24.592037 (+ 1708us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:48:24.592474 (+ 437us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:48:24.592615 (+ 141us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:48:24.595262 (+ 2647us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 14:48:24.595271 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 14:48:24.595288 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 14:48:24.595323 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0620 14:48:24.598684 (+ 3361us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:48:24.599367 (+ 683us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:48:24.599376 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:48:24.599386 (+ 10us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:48:24.599654 (+ 268us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:48:24.600394 (+ 740us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:48:24.600400 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:48:24.600414 (+ 14us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:48:24.601032 (+ 618us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 14:48:24.601361 (+ 329us) server_negotiation.cc:1035] Waiting for connection context
0620 14:48:24.601924 (+ 563us) server_negotiation.cc:299] Negotiation successful
0620 14:48:24.602357 (+ 433us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":70}
Jun 20 14:48:24 dist-test-slave-q9j6 krb5kdc[8655](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.3@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
Jun 20 14:48:24 dist-test-slave-q9j6 krb5kdc[8655](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.3@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
I20260620 14:48:24.610559 9657 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:48:24.602520 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:41725 (local address 127.0.97.3:58855)
0620 14:48:24.602683 (+ 163us) negotiation.cc:107] Waiting for socket to connect
0620 14:48:24.602702 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0620 14:48:24.602807 (+ 105us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 14:48:24.603127 (+ 320us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 14:48:24.603131 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 14:48:24.603452 (+ 321us) client_negotiation.cc:189] Negotiated authn=SASL
0620 14:48:24.603660 (+ 208us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:48:24.603673 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:48:24.604740 (+ 1067us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:48:24.604746 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:48:24.605852 (+ 1106us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:48:24.605865 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:48:24.606605 (+ 740us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:48:24.606611 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:48:24.606791 (+ 180us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:48:24.607433 (+ 642us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 14:48:24.607451 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0620 14:48:24.610361 (+ 2910us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.97.62:41725: Server oryx/127.0.97.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":54}
W20260620 14:48:24.611064 9656 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 14:48:24.602612 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.3:58855 (local address 127.0.97.62:41725)
0620 14:48:24.602832 (+ 220us) server_negotiation.cc:206] Beginning negotiation
0620 14:48:24.602838 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 14:48:24.602858 (+ 20us) server_negotiation.cc:407] Connection header received
0620 14:48:24.602911 (+ 53us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:48:24.602916 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:48:24.602985 (+ 69us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:48:24.603085 (+ 100us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:48:24.603807 (+ 722us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:48:24.604590 (+ 783us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:48:24.606011 (+ 1421us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:48:24.606455 (+ 444us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:48:24.606593 (+ 138us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:48:24.610547 (+ 3954us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.97.3:58855: BlockingRecv error: recv got EOF from 127.0.97.3:58855 (error 108)
Metrics: {"server-negotiator.queue_time_us":76}
Jun 20 14:48:24 dist-test-slave-q9j6 krb5kdc[8655](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.3@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
Jun 20 14:48:24 dist-test-slave-q9j6 krb5kdc[8655](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.3@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
I20260620 14:48:24.621557 9657 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:48:24.611237 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:41725 (local address 127.0.97.3:37563)
0620 14:48:24.611422 (+ 185us) negotiation.cc:107] Waiting for socket to connect
0620 14:48:24.611438 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0620 14:48:24.611516 (+ 78us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 14:48:24.611846 (+ 330us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 14:48:24.611851 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 14:48:24.612148 (+ 297us) client_negotiation.cc:189] Negotiated authn=SASL
0620 14:48:24.612285 (+ 137us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:48:24.612292 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:48:24.613909 (+ 1617us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:48:24.613920 (+ 11us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:48:24.614806 (+ 886us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:48:24.614816 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:48:24.615507 (+ 691us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:48:24.615513 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:48:24.615714 (+ 201us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:48:24.616351 (+ 637us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 14:48:24.616372 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0620 14:48:24.621210 (+ 4838us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.97.62:41725: Server oryx/127.0.97.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":68}
W20260620 14:48:24.622109 9653 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260620 14:48:24.622359 9656 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 14:48:24.611295 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.3:37563 (local address 127.0.97.62:41725)
0620 14:48:24.611505 (+ 210us) server_negotiation.cc:206] Beginning negotiation
0620 14:48:24.611509 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 14:48:24.611525 (+ 16us) server_negotiation.cc:407] Connection header received
0620 14:48:24.611633 (+ 108us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:48:24.611641 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:48:24.611711 (+ 70us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:48:24.611818 (+ 107us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:48:24.612421 (+ 603us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:48:24.613742 (+ 1321us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:48:24.614949 (+ 1207us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:48:24.615362 (+ 413us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:48:24.615494 (+ 132us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:48:24.622067 (+ 6573us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.97.3:37563: BlockingRecv error: recv got EOF from 127.0.97.3:37563 (error 108)
Metrics: {"server-negotiator.queue_time_us":113}
Jun 20 14:48:24 dist-test-slave-q9j6 krb5kdc[8655](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.1@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
Jun 20 14:48:24 dist-test-slave-q9j6 krb5kdc[8655](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.1@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
I20260620 14:48:24.703198 9658 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:48:24.694767 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:41725 (local address 127.0.97.1:60155)
0620 14:48:24.695329 (+ 562us) negotiation.cc:107] Waiting for socket to connect
0620 14:48:24.695349 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0620 14:48:24.695503 (+ 154us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 14:48:24.695895 (+ 392us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 14:48:24.695899 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 14:48:24.696389 (+ 490us) client_negotiation.cc:189] Negotiated authn=SASL
0620 14:48:24.696576 (+ 187us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:48:24.696585 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:48:24.697589 (+ 1004us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:48:24.697595 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:48:24.698565 (+ 970us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:48:24.698579 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:48:24.699352 (+ 773us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:48:24.699360 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:48:24.699541 (+ 181us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:48:24.700198 (+ 657us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 14:48:24.700216 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0620 14:48:24.702960 (+ 2744us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.97.62:41725: Server oryx/127.0.97.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":421,"thread_start_us":308,"threads_started":1}
W20260620 14:48:24.703372 9656 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 14:48:24.694843 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.1:60155 (local address 127.0.97.62:41725)
0620 14:48:24.695033 (+ 190us) server_negotiation.cc:206] Beginning negotiation
0620 14:48:24.695041 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0620 14:48:24.695469 (+ 428us) server_negotiation.cc:407] Connection header received
0620 14:48:24.695668 (+ 199us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:48:24.695673 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:48:24.695754 (+ 81us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:48:24.695861 (+ 107us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:48:24.696728 (+ 867us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:48:24.697438 (+ 710us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:48:24.698731 (+ 1293us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:48:24.699151 (+ 420us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:48:24.699287 (+ 136us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:48:24.703166 (+ 3879us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.97.1:60155: BlockingRecv error: recv got EOF from 127.0.97.1:60155 (error 108)
Metrics: {"server-negotiator.queue_time_us":46}
Jun 20 14:48:25 dist-test-slave-q9j6 krb5kdc[8655](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.2@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
Jun 20 14:48:25 dist-test-slave-q9j6 krb5kdc[8655](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.2@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
I20260620 14:48:25.165975 9659 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:48:25.156531 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:41725 (local address 127.0.97.2:42529)
0620 14:48:25.157108 (+ 577us) negotiation.cc:107] Waiting for socket to connect
0620 14:48:25.157133 (+ 25us) client_negotiation.cc:174] Beginning negotiation
0620 14:48:25.157275 (+ 142us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 14:48:25.157777 (+ 502us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 14:48:25.157783 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 14:48:25.158397 (+ 614us) client_negotiation.cc:189] Negotiated authn=SASL
0620 14:48:25.158736 (+ 339us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:48:25.158748 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:48:25.160274 (+ 1526us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:48:25.160278 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:48:25.161230 (+ 952us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:48:25.161241 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:48:25.162211 (+ 970us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:48:25.162217 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:48:25.162434 (+ 217us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:48:25.163030 (+ 596us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 14:48:25.163048 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0620 14:48:25.165747 (+ 2699us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.97.62:41725: Server oryx/127.0.97.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":415,"thread_start_us":259,"threads_started":1}
W20260620 14:48:25.166193 9656 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 14:48:25.157186 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.2:42529 (local address 127.0.97.62:41725)
0620 14:48:25.157377 (+ 191us) server_negotiation.cc:206] Beginning negotiation
0620 14:48:25.157383 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 14:48:25.157401 (+ 18us) server_negotiation.cc:407] Connection header received
0620 14:48:25.157484 (+ 83us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:48:25.157489 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:48:25.157584 (+ 95us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:48:25.157694 (+ 110us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:48:25.158902 (+ 1208us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:48:25.160126 (+ 1224us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:48:25.161339 (+ 1213us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:48:25.162001 (+ 662us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:48:25.162128 (+ 127us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:48:25.165956 (+ 3828us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.97.2:42529: BlockingRecv error: recv got EOF from 127.0.97.2:42529 (error 108)
Metrics: {"server-negotiator.queue_time_us":38}
Jun 20 14:48:25 dist-test-slave-q9j6 krb5kdc[8655](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.3@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
Jun 20 14:48:25 dist-test-slave-q9j6 krb5kdc[8655](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.3@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
I20260620 14:48:25.636329 9660 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:48:25.622785 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:41725 (local address 127.0.97.3:35999)
0620 14:48:25.623262 (+ 477us) negotiation.cc:107] Waiting for socket to connect
0620 14:48:25.623277 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0620 14:48:25.623437 (+ 160us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 14:48:25.623853 (+ 416us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 14:48:25.623858 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 14:48:25.624314 (+ 456us) client_negotiation.cc:189] Negotiated authn=SASL
0620 14:48:25.624516 (+ 202us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:48:25.624525 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:48:25.625569 (+ 1044us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:48:25.625576 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:48:25.626525 (+ 949us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:48:25.626536 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:48:25.627323 (+ 787us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:48:25.627329 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:48:25.627504 (+ 175us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:48:25.633113 (+ 5609us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 14:48:25.633134 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0620 14:48:25.635997 (+ 2863us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.97.62:41725: Server oryx/127.0.97.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":381,"thread_start_us":255,"threads_started":1}
W20260620 14:48:25.636471 9656 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 14:48:25.622854 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.3:35999 (local address 127.0.97.62:41725)
0620 14:48:25.623054 (+ 200us) server_negotiation.cc:206] Beginning negotiation
0620 14:48:25.623063 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0620 14:48:25.623369 (+ 306us) server_negotiation.cc:407] Connection header received
0620 14:48:25.623569 (+ 200us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:48:25.623574 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:48:25.623710 (+ 136us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:48:25.623820 (+ 110us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:48:25.624668 (+ 848us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:48:25.625429 (+ 761us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:48:25.626669 (+ 1240us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:48:25.627127 (+ 458us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:48:25.627275 (+ 148us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:48:25.636227 (+ 8952us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.97.3:35999: BlockingRecv error: recv got EOF from 127.0.97.3:35999 (error 108)
Metrics: {"server-negotiator.queue_time_us":55}
Jun 20 14:48:25 dist-test-slave-q9j6 krb5kdc[8655](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.1@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
Jun 20 14:48:25 dist-test-slave-q9j6 krb5kdc[8655](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.1@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
I20260620 14:48:25.712589 9661 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:48:25.704093 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:41725 (local address 127.0.97.1:54715)
0620 14:48:25.704615 (+ 522us) negotiation.cc:107] Waiting for socket to connect
0620 14:48:25.704631 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0620 14:48:25.704791 (+ 160us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 14:48:25.705127 (+ 336us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 14:48:25.705131 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 14:48:25.705556 (+ 425us) client_negotiation.cc:189] Negotiated authn=SASL
0620 14:48:25.705698 (+ 142us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:48:25.705706 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:48:25.706707 (+ 1001us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:48:25.706713 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:48:25.707716 (+ 1003us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:48:25.707728 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:48:25.708468 (+ 740us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:48:25.708474 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:48:25.708645 (+ 171us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:48:25.709220 (+ 575us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 14:48:25.709237 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0620 14:48:25.712220 (+ 2983us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.97.62:41725: Server oryx/127.0.97.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":429,"thread_start_us":302,"threads_started":1}
W20260620 14:48:25.712692 9656 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 14:48:25.704142 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.1:54715 (local address 127.0.97.62:41725)
0620 14:48:25.704389 (+ 247us) server_negotiation.cc:206] Beginning negotiation
0620 14:48:25.704398 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0620 14:48:25.704717 (+ 319us) server_negotiation.cc:407] Connection header received
0620 14:48:25.704935 (+ 218us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:48:25.704938 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:48:25.704996 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:48:25.705097 (+ 101us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:48:25.705849 (+ 752us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:48:25.706561 (+ 712us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:48:25.707850 (+ 1289us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:48:25.708270 (+ 420us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:48:25.708404 (+ 134us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:48:25.712429 (+ 4025us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.97.1:54715: BlockingRecv error: recv got EOF from 127.0.97.1:54715 (error 108)
Metrics: {"server-negotiator.queue_time_us":55}
Jun 20 14:48:26 dist-test-slave-q9j6 krb5kdc[8655](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.2@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
Jun 20 14:48:26 dist-test-slave-q9j6 krb5kdc[8655](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.2@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
I20260620 14:48:26.175779 9662 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:48:26.167165 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:41725 (local address 127.0.97.2:39963)
0620 14:48:26.167713 (+ 548us) negotiation.cc:107] Waiting for socket to connect
0620 14:48:26.167730 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0620 14:48:26.167874 (+ 144us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 14:48:26.168152 (+ 278us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 14:48:26.168155 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 14:48:26.168525 (+ 370us) client_negotiation.cc:189] Negotiated authn=SASL
0620 14:48:26.168673 (+ 148us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:48:26.168682 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:48:26.170070 (+ 1388us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:48:26.170082 (+ 12us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:48:26.171125 (+ 1043us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:48:26.171138 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:48:26.171900 (+ 762us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:48:26.171904 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:48:26.172019 (+ 115us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:48:26.172623 (+ 604us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 14:48:26.172641 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0620 14:48:26.175456 (+ 2815us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.97.62:41725: Server oryx/127.0.97.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":380,"thread_start_us":271,"threads_started":1}
W20260620 14:48:26.175959 9656 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 14:48:26.167236 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.2:39963 (local address 127.0.97.62:41725)
0620 14:48:26.167499 (+ 263us) server_negotiation.cc:206] Beginning negotiation
0620 14:48:26.167506 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0620 14:48:26.167819 (+ 313us) server_negotiation.cc:407] Connection header received
0620 14:48:26.167965 (+ 146us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:48:26.167968 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:48:26.168028 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:48:26.168116 (+ 88us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:48:26.168854 (+ 738us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:48:26.169905 (+ 1051us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:48:26.171265 (+ 1360us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:48:26.171760 (+ 495us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:48:26.171860 (+ 100us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:48:26.175734 (+ 3874us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.97.2:39963: BlockingRecv error: recv got EOF from 127.0.97.2:39963 (error 108)
Metrics: {"server-negotiator.queue_time_us":65}
Jun 20 14:48:26 dist-test-slave-q9j6 krb5kdc[8655](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.3@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
Jun 20 14:48:26 dist-test-slave-q9j6 krb5kdc[8655](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.3@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
I20260620 14:48:26.645408 9663 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:48:26.637247 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:41725 (local address 127.0.97.3:34123)
0620 14:48:26.637799 (+ 552us) negotiation.cc:107] Waiting for socket to connect
0620 14:48:26.637813 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0620 14:48:26.637978 (+ 165us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 14:48:26.638339 (+ 361us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 14:48:26.638344 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 14:48:26.638747 (+ 403us) client_negotiation.cc:189] Negotiated authn=SASL
0620 14:48:26.638895 (+ 148us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:48:26.638903 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:48:26.639976 (+ 1073us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:48:26.639981 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:48:26.640872 (+ 891us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:48:26.640883 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:48:26.641599 (+ 716us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:48:26.641605 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:48:26.641773 (+ 168us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:48:26.642396 (+ 623us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 14:48:26.642414 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0620 14:48:26.645177 (+ 2763us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.97.62:41725: Server oryx/127.0.97.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":434,"thread_start_us":306,"threads_started":1}
W20260620 14:48:26.645655 9656 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 14:48:26.637460 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.3:34123 (local address 127.0.97.62:41725)
0620 14:48:26.637678 (+ 218us) server_negotiation.cc:206] Beginning negotiation
0620 14:48:26.637685 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0620 14:48:26.637896 (+ 211us) server_negotiation.cc:407] Connection header received
0620 14:48:26.638116 (+ 220us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:48:26.638121 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:48:26.638205 (+ 84us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:48:26.638314 (+ 109us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:48:26.639032 (+ 718us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:48:26.639844 (+ 812us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:48:26.641019 (+ 1175us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:48:26.641434 (+ 415us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:48:26.641579 (+ 145us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:48:26.645390 (+ 3811us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.97.3:34123: BlockingRecv error: recv got EOF from 127.0.97.3:34123 (error 108)
Metrics: {"server-negotiator.queue_time_us":82}
Jun 20 14:48:26 dist-test-slave-q9j6 krb5kdc[8655](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.1@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
Jun 20 14:48:26 dist-test-slave-q9j6 krb5kdc[8655](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.1@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
I20260620 14:48:26.722259 9664 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:48:26.713674 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:41725 (local address 127.0.97.1:33445)
0620 14:48:26.714129 (+ 455us) negotiation.cc:107] Waiting for socket to connect
0620 14:48:26.714146 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0620 14:48:26.714266 (+ 120us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 14:48:26.714547 (+ 281us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 14:48:26.714551 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 14:48:26.714893 (+ 342us) client_negotiation.cc:189] Negotiated authn=SASL
0620 14:48:26.715068 (+ 175us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:48:26.715077 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:48:26.716135 (+ 1058us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:48:26.716140 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:48:26.717071 (+ 931us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:48:26.717085 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:48:26.717744 (+ 659us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:48:26.717748 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:48:26.717894 (+ 146us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:48:26.718523 (+ 629us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 14:48:26.718540 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0620 14:48:26.721873 (+ 3333us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.97.62:41725: Server oryx/127.0.97.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":360,"thread_start_us":247,"threads_started":1}
W20260620 14:48:26.722362 9656 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 14:48:26.713743 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.1:33445 (local address 127.0.97.62:41725)
0620 14:48:26.713985 (+ 242us) server_negotiation.cc:206] Beginning negotiation
0620 14:48:26.713992 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0620 14:48:26.714230 (+ 238us) server_negotiation.cc:407] Connection header received
0620 14:48:26.714352 (+ 122us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:48:26.714355 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:48:26.714416 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:48:26.714504 (+ 88us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:48:26.715195 (+ 691us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:48:26.715994 (+ 799us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:48:26.717224 (+ 1230us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:48:26.717610 (+ 386us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:48:26.717709 (+ 99us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:48:26.722110 (+ 4401us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.97.1:33445: BlockingRecv error: recv got EOF from 127.0.97.1:33445 (error 108)
Metrics: {"server-negotiator.queue_time_us":70}
Jun 20 14:48:27 dist-test-slave-q9j6 krb5kdc[8655](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.2@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
Jun 20 14:48:27 dist-test-slave-q9j6 krb5kdc[8655](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.2@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
I20260620 14:48:27.184818 9665 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:48:27.176727 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:41725 (local address 127.0.97.2:47353)
0620 14:48:27.177378 (+ 651us) negotiation.cc:107] Waiting for socket to connect
0620 14:48:27.177394 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0620 14:48:27.177552 (+ 158us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 14:48:27.177874 (+ 322us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 14:48:27.177877 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 14:48:27.178199 (+ 322us) client_negotiation.cc:189] Negotiated authn=SASL
0620 14:48:27.178360 (+ 161us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:48:27.178379 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:48:27.179353 (+ 974us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:48:27.179357 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:48:27.180417 (+ 1060us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:48:27.180430 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:48:27.181071 (+ 641us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:48:27.181075 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:48:27.181204 (+ 129us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:48:27.181797 (+ 593us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 14:48:27.181814 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0620 14:48:27.184570 (+ 2756us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.97.62:41725: Server oryx/127.0.97.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":535,"thread_start_us":420,"threads_started":1}
W20260620 14:48:27.185063 9656 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 14:48:27.176789 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.2:47353 (local address 127.0.97.62:41725)
0620 14:48:27.177039 (+ 250us) server_negotiation.cc:206] Beginning negotiation
0620 14:48:27.177049 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0620 14:48:27.177477 (+ 428us) server_negotiation.cc:407] Connection header received
0620 14:48:27.177681 (+ 204us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:48:27.177684 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:48:27.177745 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:48:27.177827 (+ 82us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:48:27.178495 (+ 668us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:48:27.179236 (+ 741us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:48:27.180539 (+ 1303us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:48:27.180943 (+ 404us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:48:27.181037 (+ 94us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:48:27.184754 (+ 3717us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.97.2:47353: BlockingRecv error: recv got EOF from 127.0.97.2:47353 (error 108)
Metrics: {"server-negotiator.queue_time_us":69}
Jun 20 14:48:27 dist-test-slave-q9j6 krb5kdc[8655](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.3@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
Jun 20 14:48:27 dist-test-slave-q9j6 krb5kdc[8655](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.3@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
I20260620 14:48:27.655329 9666 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:48:27.646547 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:41725 (local address 127.0.97.3:41385)
0620 14:48:27.647127 (+ 580us) negotiation.cc:107] Waiting for socket to connect
0620 14:48:27.647151 (+ 24us) client_negotiation.cc:174] Beginning negotiation
0620 14:48:27.647277 (+ 126us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 14:48:27.647708 (+ 431us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 14:48:27.647711 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 14:48:27.648075 (+ 364us) client_negotiation.cc:189] Negotiated authn=SASL
0620 14:48:27.648243 (+ 168us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:48:27.648252 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:48:27.649301 (+ 1049us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:48:27.649305 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:48:27.650207 (+ 902us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:48:27.650219 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:48:27.650883 (+ 664us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:48:27.650887 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:48:27.651010 (+ 123us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:48:27.651673 (+ 663us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 14:48:27.651693 (+ 20us) client_negotiation.cc:656] calling sasl_client_start()
0620 14:48:27.654986 (+ 3293us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.97.62:41725: Server oryx/127.0.97.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":451,"thread_start_us":339,"threads_started":1}
W20260620 14:48:27.655426 9656 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 14:48:27.646659 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.3:41385 (local address 127.0.97.62:41725)
0620 14:48:27.646868 (+ 209us) server_negotiation.cc:206] Beginning negotiation
0620 14:48:27.646872 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 14:48:27.647240 (+ 368us) server_negotiation.cc:407] Connection header received
0620 14:48:27.647422 (+ 182us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:48:27.647426 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:48:27.647509 (+ 83us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:48:27.647663 (+ 154us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:48:27.648382 (+ 719us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:48:27.649149 (+ 767us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:48:27.650334 (+ 1185us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:48:27.650753 (+ 419us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:48:27.650849 (+ 96us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:48:27.655210 (+ 4361us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.97.3:41385: BlockingRecv error: recv got EOF from 127.0.97.3:41385 (error 108)
Metrics: {"server-negotiator.queue_time_us":94}
Jun 20 14:48:27 dist-test-slave-q9j6 krb5kdc[8655](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.1@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
Jun 20 14:48:27 dist-test-slave-q9j6 krb5kdc[8655](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.1@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
I20260620 14:48:27.731995 9667 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:48:27.723323 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:41725 (local address 127.0.97.1:42935)
0620 14:48:27.723895 (+ 572us) negotiation.cc:107] Waiting for socket to connect
0620 14:48:27.723921 (+ 26us) client_negotiation.cc:174] Beginning negotiation
0620 14:48:27.724059 (+ 138us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 14:48:27.724358 (+ 299us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 14:48:27.724361 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 14:48:27.724721 (+ 360us) client_negotiation.cc:189] Negotiated authn=SASL
0620 14:48:27.724874 (+ 153us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:48:27.724883 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:48:27.725876 (+ 993us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:48:27.725880 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:48:27.726899 (+ 1019us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:48:27.726918 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:48:27.727623 (+ 705us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:48:27.727629 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:48:27.728051 (+ 422us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:48:27.728801 (+ 750us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 14:48:27.728823 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0620 14:48:27.731760 (+ 2937us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.97.62:41725: Server oryx/127.0.97.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":456,"thread_start_us":352,"threads_started":1}
W20260620 14:48:27.732203 9656 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 14:48:27.723456 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.1:42935 (local address 127.0.97.62:41725)
0620 14:48:27.723702 (+ 246us) server_negotiation.cc:206] Beginning negotiation
0620 14:48:27.723709 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0620 14:48:27.724004 (+ 295us) server_negotiation.cc:407] Connection header received
0620 14:48:27.724152 (+ 148us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:48:27.724155 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:48:27.724218 (+ 63us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:48:27.724315 (+ 97us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:48:27.725010 (+ 695us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:48:27.725752 (+ 742us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:48:27.727059 (+ 1307us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:48:27.727465 (+ 406us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:48:27.727565 (+ 100us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:48:27.731967 (+ 4402us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.97.1:42935: BlockingRecv error: recv got EOF from 127.0.97.1:42935 (error 108)
Metrics: {"server-negotiator.queue_time_us":73}
Jun 20 14:48:28 dist-test-slave-q9j6 krb5kdc[8655](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.2@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
Jun 20 14:48:28 dist-test-slave-q9j6 krb5kdc[8655](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.2@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
I20260620 14:48:28.194654 9668 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:48:28.185876 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:41725 (local address 127.0.97.2:50445)
0620 14:48:28.186405 (+ 529us) negotiation.cc:107] Waiting for socket to connect
0620 14:48:28.186429 (+ 24us) client_negotiation.cc:174] Beginning negotiation
0620 14:48:28.186599 (+ 170us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 14:48:28.186966 (+ 367us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 14:48:28.186973 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 14:48:28.187349 (+ 376us) client_negotiation.cc:189] Negotiated authn=SASL
0620 14:48:28.187495 (+ 146us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:48:28.187503 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:48:28.188545 (+ 1042us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:48:28.188552 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:48:28.189553 (+ 1001us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:48:28.189566 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:48:28.190266 (+ 700us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:48:28.190271 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:48:28.190461 (+ 190us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:48:28.191106 (+ 645us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 14:48:28.191124 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0620 14:48:28.194305 (+ 3181us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.97.62:41725: Server oryx/127.0.97.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":396,"thread_start_us":274,"threads_started":1}
W20260620 14:48:28.194715 9656 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 14:48:28.185926 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.2:50445 (local address 127.0.97.62:41725)
0620 14:48:28.186171 (+ 245us) server_negotiation.cc:206] Beginning negotiation
0620 14:48:28.186178 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0620 14:48:28.186533 (+ 355us) server_negotiation.cc:407] Connection header received
0620 14:48:28.186735 (+ 202us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:48:28.186738 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:48:28.186798 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:48:28.186922 (+ 124us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:48:28.187675 (+ 753us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:48:28.188405 (+ 730us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:48:28.189703 (+ 1298us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:48:28.190114 (+ 411us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:48:28.190249 (+ 135us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:48:28.194503 (+ 4254us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.97.2:50445: BlockingRecv error: recv got EOF from 127.0.97.2:50445 (error 108)
Metrics: {"server-negotiator.queue_time_us":61}
Jun 20 14:48:28 dist-test-slave-q9j6 krb5kdc[8655](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.3@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
Jun 20 14:48:28 dist-test-slave-q9j6 krb5kdc[8655](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.3@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
I20260620 14:48:28.665367 9669 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:48:28.656587 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:41725 (local address 127.0.97.3:53885)
0620 14:48:28.657096 (+ 509us) negotiation.cc:107] Waiting for socket to connect
0620 14:48:28.657117 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0620 14:48:28.657263 (+ 146us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 14:48:28.657678 (+ 415us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 14:48:28.657683 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 14:48:28.658084 (+ 401us) client_negotiation.cc:189] Negotiated authn=SASL
0620 14:48:28.658266 (+ 182us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:48:28.658278 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:48:28.659557 (+ 1279us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:48:28.659562 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:48:28.660624 (+ 1062us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:48:28.660636 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:48:28.661366 (+ 730us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:48:28.661370 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:48:28.661524 (+ 154us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:48:28.662099 (+ 575us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 14:48:28.662117 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0620 14:48:28.665029 (+ 2912us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.97.62:41725: Server oryx/127.0.97.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":385,"thread_start_us":277,"threads_started":1}
W20260620 14:48:28.665503 9656 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 14:48:28.656682 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.3:53885 (local address 127.0.97.62:41725)
0620 14:48:28.656926 (+ 244us) server_negotiation.cc:206] Beginning negotiation
0620 14:48:28.656933 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0620 14:48:28.657221 (+ 288us) server_negotiation.cc:407] Connection header received
0620 14:48:28.657404 (+ 183us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:48:28.657409 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:48:28.657516 (+ 107us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:48:28.657634 (+ 118us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:48:28.658480 (+ 846us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:48:28.659396 (+ 916us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:48:28.660748 (+ 1352us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:48:28.661232 (+ 484us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:48:28.661335 (+ 103us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:48:28.665224 (+ 3889us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.97.3:53885: BlockingRecv error: recv got EOF from 127.0.97.3:53885 (error 108)
Metrics: {"server-negotiator.queue_time_us":76}
Jun 20 14:48:28 dist-test-slave-q9j6 krb5kdc[8655](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.1@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
Jun 20 14:48:28 dist-test-slave-q9j6 krb5kdc[8655](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.1@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
I20260620 14:48:28.741585 9670 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:48:28.733012 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:41725 (local address 127.0.97.1:39453)
0620 14:48:28.733592 (+ 580us) negotiation.cc:107] Waiting for socket to connect
0620 14:48:28.733609 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0620 14:48:28.733754 (+ 145us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 14:48:28.734106 (+ 352us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 14:48:28.734111 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 14:48:28.734544 (+ 433us) client_negotiation.cc:189] Negotiated authn=SASL
0620 14:48:28.734713 (+ 169us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:48:28.734722 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:48:28.735774 (+ 1052us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:48:28.735780 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:48:28.736695 (+ 915us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:48:28.736707 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:48:28.737440 (+ 733us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:48:28.737446 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:48:28.737622 (+ 176us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:48:28.738307 (+ 685us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 14:48:28.738325 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0620 14:48:28.741233 (+ 2908us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.97.62:41725: Server oryx/127.0.97.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":454,"thread_start_us":255,"threads_started":1}
W20260620 14:48:28.741700 9656 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 14:48:28.733069 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.1:39453 (local address 127.0.97.62:41725)
0620 14:48:28.733281 (+ 212us) server_negotiation.cc:206] Beginning negotiation
0620 14:48:28.733289 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0620 14:48:28.733690 (+ 401us) server_negotiation.cc:407] Connection header received
0620 14:48:28.733889 (+ 199us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:48:28.733894 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:48:28.733967 (+ 73us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:48:28.734069 (+ 102us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:48:28.734869 (+ 800us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:48:28.735571 (+ 702us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:48:28.736855 (+ 1284us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:48:28.737282 (+ 427us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:48:28.737413 (+ 131us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:48:28.741469 (+ 4056us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.97.1:39453: BlockingRecv error: recv got EOF from 127.0.97.1:39453 (error 108)
Metrics: {"server-negotiator.queue_time_us":63}
Jun 20 14:48:29 dist-test-slave-q9j6 krb5kdc[8655](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.2@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
Jun 20 14:48:29 dist-test-slave-q9j6 krb5kdc[8655](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.2@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
I20260620 14:48:29.204348 9671 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:48:29.195845 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:41725 (local address 127.0.97.2:59025)
0620 14:48:29.196421 (+ 576us) negotiation.cc:107] Waiting for socket to connect
0620 14:48:29.196437 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0620 14:48:29.196561 (+ 124us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 14:48:29.196867 (+ 306us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 14:48:29.196871 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 14:48:29.197254 (+ 383us) client_negotiation.cc:189] Negotiated authn=SASL
0620 14:48:29.197412 (+ 158us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:48:29.197421 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:48:29.198478 (+ 1057us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:48:29.198481 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:48:29.199398 (+ 917us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:48:29.199409 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:48:29.200081 (+ 672us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:48:29.200086 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:48:29.200206 (+ 120us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:48:29.200781 (+ 575us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 14:48:29.200798 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0620 14:48:29.203997 (+ 3199us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.97.62:41725: Server oryx/127.0.97.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":457,"thread_start_us":302,"threads_started":1}
W20260620 14:48:29.204419 9656 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 14:48:29.195994 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.2:59025 (local address 127.0.97.62:41725)
0620 14:48:29.196167 (+ 173us) server_negotiation.cc:206] Beginning negotiation
0620 14:48:29.196171 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 14:48:29.196514 (+ 343us) server_negotiation.cc:407] Connection header received
0620 14:48:29.196666 (+ 152us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:48:29.196669 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:48:29.196728 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:48:29.196817 (+ 89us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:48:29.197547 (+ 730us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:48:29.198340 (+ 793us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:48:29.199529 (+ 1189us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:48:29.199955 (+ 426us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:48:29.200051 (+ 96us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:48:29.204197 (+ 4146us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.97.2:59025: BlockingRecv error: recv got EOF from 127.0.97.2:59025 (error 108)
Metrics: {"server-negotiator.queue_time_us":60}
Jun 20 14:48:29 dist-test-slave-q9j6 krb5kdc[8655](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.3@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
Jun 20 14:48:29 dist-test-slave-q9j6 krb5kdc[8655](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.3@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
I20260620 14:48:29.675397 9672 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:48:29.666610 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:41725 (local address 127.0.97.3:44651)
0620 14:48:29.667157 (+ 547us) negotiation.cc:107] Waiting for socket to connect
0620 14:48:29.667175 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0620 14:48:29.667287 (+ 112us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 14:48:29.667675 (+ 388us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 14:48:29.667679 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 14:48:29.668209 (+ 530us) client_negotiation.cc:189] Negotiated authn=SASL
0620 14:48:29.668383 (+ 174us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:48:29.668392 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:48:29.669395 (+ 1003us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:48:29.669398 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:48:29.670311 (+ 913us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:48:29.670323 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:48:29.671052 (+ 729us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:48:29.671059 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:48:29.671193 (+ 134us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:48:29.671974 (+ 781us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 14:48:29.672001 (+ 27us) client_negotiation.cc:656] calling sasl_client_start()
0620 14:48:29.675142 (+ 3141us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.97.62:41725: Server oryx/127.0.97.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":437,"thread_start_us":282,"threads_started":1}
W20260620 14:48:29.675659 9656 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 14:48:29.666693 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.3:44651 (local address 127.0.97.62:41725)
0620 14:48:29.666885 (+ 192us) server_negotiation.cc:206] Beginning negotiation
0620 14:48:29.666890 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 14:48:29.667252 (+ 362us) server_negotiation.cc:407] Connection header received
0620 14:48:29.667385 (+ 133us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:48:29.667389 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:48:29.667474 (+ 85us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:48:29.667570 (+ 96us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:48:29.668536 (+ 966us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:48:29.669268 (+ 732us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:48:29.670449 (+ 1181us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:48:29.670872 (+ 423us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:48:29.670975 (+ 103us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:48:29.675331 (+ 4356us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.97.3:44651: BlockingRecv error: recv got EOF from 127.0.97.3:44651 (error 108)
Metrics: {"server-negotiator.queue_time_us":74}
Jun 20 14:48:29 dist-test-slave-q9j6 krb5kdc[8655](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.1@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
Jun 20 14:48:29 dist-test-slave-q9j6 krb5kdc[8655](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.1@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
I20260620 14:48:29.751432 9673 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:48:29.742865 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:41725 (local address 127.0.97.1:48013)
0620 14:48:29.743454 (+ 589us) negotiation.cc:107] Waiting for socket to connect
0620 14:48:29.743470 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0620 14:48:29.743591 (+ 121us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 14:48:29.743981 (+ 390us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 14:48:29.743985 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 14:48:29.744359 (+ 374us) client_negotiation.cc:189] Negotiated authn=SASL
0620 14:48:29.744508 (+ 149us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:48:29.744518 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:48:29.745920 (+ 1402us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:48:29.745926 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:48:29.746820 (+ 894us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:48:29.746833 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:48:29.747571 (+ 738us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:48:29.747575 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:48:29.747770 (+ 195us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:48:29.748415 (+ 645us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 14:48:29.748442 (+ 27us) client_negotiation.cc:656] calling sasl_client_start()
0620 14:48:29.751204 (+ 2762us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.97.62:41725: Server oryx/127.0.97.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":496,"thread_start_us":400,"threads_started":1}
W20260620 14:48:29.751695 9656 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 14:48:29.743114 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.1:48013 (local address 127.0.97.62:41725)
0620 14:48:29.743457 (+ 343us) server_negotiation.cc:206] Beginning negotiation
0620 14:48:29.743464 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0620 14:48:29.743557 (+ 93us) server_negotiation.cc:407] Connection header received
0620 14:48:29.743717 (+ 160us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:48:29.743721 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:48:29.743804 (+ 83us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:48:29.743919 (+ 115us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:48:29.744688 (+ 769us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:48:29.745753 (+ 1065us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:48:29.746969 (+ 1216us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:48:29.747423 (+ 454us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:48:29.747529 (+ 106us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:48:29.751432 (+ 3903us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.97.1:48013: BlockingRecv error: recv got EOF from 127.0.97.1:48013 (error 108)
Metrics: {"server-negotiator.queue_time_us":187}
Jun 20 14:48:30 dist-test-slave-q9j6 krb5kdc[8655](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.2@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
Jun 20 14:48:30 dist-test-slave-q9j6 krb5kdc[8655](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.2@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
I20260620 14:48:30.213712 9674 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:48:30.205325 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:41725 (local address 127.0.97.2:48837)
0620 14:48:30.205851 (+ 526us) negotiation.cc:107] Waiting for socket to connect
0620 14:48:30.205868 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0620 14:48:30.206022 (+ 154us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 14:48:30.206353 (+ 331us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 14:48:30.206358 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 14:48:30.206778 (+ 420us) client_negotiation.cc:189] Negotiated authn=SASL
0620 14:48:30.206922 (+ 144us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:48:30.206931 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:48:30.207967 (+ 1036us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:48:30.207978 (+ 11us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:48:30.208984 (+ 1006us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:48:30.208996 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:48:30.209765 (+ 769us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:48:30.209780 (+ 15us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:48:30.209940 (+ 160us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:48:30.210527 (+ 587us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 14:48:30.210544 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0620 14:48:30.213479 (+ 2935us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.97.62:41725: Server oryx/127.0.97.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":433,"thread_start_us":311,"threads_started":1}
W20260620 14:48:30.213909 9656 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 14:48:30.205405 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.2:48837 (local address 127.0.97.62:41725)
0620 14:48:30.205616 (+ 211us) server_negotiation.cc:206] Beginning negotiation
0620 14:48:30.205623 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0620 14:48:30.205954 (+ 331us) server_negotiation.cc:407] Connection header received
0620 14:48:30.206152 (+ 198us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:48:30.206156 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:48:30.206216 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:48:30.206328 (+ 112us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:48:30.207061 (+ 733us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:48:30.207831 (+ 770us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:48:30.209121 (+ 1290us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:48:30.209589 (+ 468us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:48:30.209724 (+ 135us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:48:30.213694 (+ 3970us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.97.2:48837: BlockingRecv error: recv got EOF from 127.0.97.2:48837 (error 108)
Metrics: {"server-negotiator.queue_time_us":71}
Jun 20 14:48:30 dist-test-slave-q9j6 krb5kdc[8655](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.3@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
Jun 20 14:48:30 dist-test-slave-q9j6 krb5kdc[8655](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.3@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
I20260620 14:48:30.688333 9675 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:48:30.676362 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:41725 (local address 127.0.97.3:37167)
0620 14:48:30.676901 (+ 539us) negotiation.cc:107] Waiting for socket to connect
0620 14:48:30.676916 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0620 14:48:30.677083 (+ 167us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 14:48:30.677439 (+ 356us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 14:48:30.677444 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 14:48:30.677837 (+ 393us) client_negotiation.cc:189] Negotiated authn=SASL
0620 14:48:30.677987 (+ 150us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:48:30.677996 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:48:30.679058 (+ 1062us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:48:30.679065 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:48:30.680065 (+ 1000us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:48:30.680078 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:48:30.680895 (+ 817us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:48:30.680902 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:48:30.681077 (+ 175us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:48:30.685095 (+ 4018us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 14:48:30.685115 (+ 20us) client_negotiation.cc:656] calling sasl_client_start()
0620 14:48:30.688117 (+ 3002us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.97.62:41725: Server oryx/127.0.97.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":448,"thread_start_us":290,"threads_started":1}
W20260620 14:48:30.688620 9656 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 14:48:30.676449 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.3:37167 (local address 127.0.97.62:41725)
0620 14:48:30.676706 (+ 257us) server_negotiation.cc:206] Beginning negotiation
0620 14:48:30.676714 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0620 14:48:30.677012 (+ 298us) server_negotiation.cc:407] Connection header received
0620 14:48:30.677210 (+ 198us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:48:30.677222 (+ 12us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:48:30.677295 (+ 73us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:48:30.677401 (+ 106us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:48:30.678137 (+ 736us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:48:30.678903 (+ 766us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:48:30.680230 (+ 1327us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:48:30.680727 (+ 497us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:48:30.680862 (+ 135us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:48:30.688333 (+ 7471us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.97.3:37167: BlockingRecv error: recv got EOF from 127.0.97.3:37167 (error 108)
Metrics: {"server-negotiator.queue_time_us":103}
Jun 20 14:48:30 dist-test-slave-q9j6 krb5kdc[8655](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.1@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
Jun 20 14:48:30 dist-test-slave-q9j6 krb5kdc[8655](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.1@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
I20260620 14:48:30.761299 9676 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:48:30.752537 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:41725 (local address 127.0.97.1:60505)
0620 14:48:30.753167 (+ 630us) negotiation.cc:107] Waiting for socket to connect
0620 14:48:30.753184 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0620 14:48:30.753303 (+ 119us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 14:48:30.753615 (+ 312us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 14:48:30.753618 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 14:48:30.754031 (+ 413us) client_negotiation.cc:189] Negotiated authn=SASL
0620 14:48:30.754197 (+ 166us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:48:30.754205 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:48:30.755222 (+ 1017us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:48:30.755226 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:48:30.756435 (+ 1209us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:48:30.756456 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:48:30.757315 (+ 859us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:48:30.757322 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:48:30.757490 (+ 168us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:48:30.758102 (+ 612us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 14:48:30.758119 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0620 14:48:30.761063 (+ 2944us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.97.62:41725: Server oryx/127.0.97.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":490,"thread_start_us":267,"threads_started":1}
W20260620 14:48:30.761475 9656 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 14:48:30.752637 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.1:60505 (local address 127.0.97.62:41725)
0620 14:48:30.752829 (+ 192us) server_negotiation.cc:206] Beginning negotiation
0620 14:48:30.752834 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 14:48:30.753274 (+ 440us) server_negotiation.cc:407] Connection header received
0620 14:48:30.753405 (+ 131us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:48:30.753409 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:48:30.753471 (+ 62us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:48:30.753565 (+ 94us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:48:30.754324 (+ 759us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:48:30.755087 (+ 763us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:48:30.756580 (+ 1493us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:48:30.757147 (+ 567us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:48:30.757269 (+ 122us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:48:30.761246 (+ 3977us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.97.1:60505: BlockingRecv error: recv got EOF from 127.0.97.1:60505 (error 108)
Metrics: {"server-negotiator.queue_time_us":80}
Jun 20 14:48:31 dist-test-slave-q9j6 krb5kdc[8655](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.2@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
Jun 20 14:48:31 dist-test-slave-q9j6 krb5kdc[8655](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.2@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
I20260620 14:48:31.223366 9677 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:48:31.214740 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:41725 (local address 127.0.97.2:37605)
0620 14:48:31.215318 (+ 578us) negotiation.cc:107] Waiting for socket to connect
0620 14:48:31.215334 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0620 14:48:31.215455 (+ 121us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 14:48:31.215796 (+ 341us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 14:48:31.215800 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 14:48:31.216171 (+ 371us) client_negotiation.cc:189] Negotiated authn=SASL
0620 14:48:31.216337 (+ 166us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:48:31.216345 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:48:31.217274 (+ 929us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:48:31.217277 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:48:31.218179 (+ 902us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:48:31.218190 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:48:31.219117 (+ 927us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:48:31.219121 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:48:31.219263 (+ 142us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:48:31.220017 (+ 754us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 14:48:31.220042 (+ 25us) client_negotiation.cc:656] calling sasl_client_start()
0620 14:48:31.222996 (+ 2954us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.97.62:41725: Server oryx/127.0.97.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":446,"thread_start_us":278,"threads_started":1}
W20260620 14:48:31.223417 9656 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 14:48:31.214861 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.2:37605 (local address 127.0.97.62:41725)
0620 14:48:31.215069 (+ 208us) server_negotiation.cc:206] Beginning negotiation
0620 14:48:31.215074 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 14:48:31.215410 (+ 336us) server_negotiation.cc:407] Connection header received
0620 14:48:31.215558 (+ 148us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:48:31.215562 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:48:31.215671 (+ 109us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:48:31.215753 (+ 82us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:48:31.216456 (+ 703us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:48:31.217155 (+ 699us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:48:31.218294 (+ 1139us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:48:31.218867 (+ 573us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:48:31.218958 (+ 91us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:48:31.223188 (+ 4230us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.97.2:37605: BlockingRecv error: recv got EOF from 127.0.97.2:37605 (error 108)
Metrics: {"server-negotiator.queue_time_us":100}
Jun 20 14:48:31 dist-test-slave-q9j6 krb5kdc[8655](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.3@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
Jun 20 14:48:31 dist-test-slave-q9j6 krb5kdc[8655](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.3@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
I20260620 14:48:31.697700 9678 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:48:31.689376 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:41725 (local address 127.0.97.3:51997)
0620 14:48:31.689998 (+ 622us) negotiation.cc:107] Waiting for socket to connect
0620 14:48:31.690014 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0620 14:48:31.690131 (+ 117us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 14:48:31.690420 (+ 289us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 14:48:31.690424 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 14:48:31.690835 (+ 411us) client_negotiation.cc:189] Negotiated authn=SASL
0620 14:48:31.691010 (+ 175us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:48:31.691019 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:48:31.692005 (+ 986us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:48:31.692009 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:48:31.692952 (+ 943us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:48:31.692964 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:48:31.693619 (+ 655us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:48:31.693623 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:48:31.693734 (+ 111us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:48:31.694331 (+ 597us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 14:48:31.694348 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0620 14:48:31.697343 (+ 2995us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.97.62:41725: Server oryx/127.0.97.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":483,"thread_start_us":284,"threads_started":1}
W20260620 14:48:31.697803 9656 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 14:48:31.689497 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.3:51997 (local address 127.0.97.62:41725)
0620 14:48:31.689668 (+ 171us) server_negotiation.cc:206] Beginning negotiation
0620 14:48:31.689673 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 14:48:31.690086 (+ 413us) server_negotiation.cc:407] Connection header received
0620 14:48:31.690226 (+ 140us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:48:31.690229 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:48:31.690286 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:48:31.690370 (+ 84us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:48:31.691134 (+ 764us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:48:31.691872 (+ 738us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:48:31.693079 (+ 1207us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:48:31.693480 (+ 401us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:48:31.693570 (+ 90us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:48:31.697556 (+ 3986us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.97.3:51997: BlockingRecv error: recv got EOF from 127.0.97.3:51997 (error 108)
Metrics: {"server-negotiator.queue_time_us":60}
Jun 20 14:48:31 dist-test-slave-q9j6 krb5kdc[8655](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.1@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
Jun 20 14:48:31 dist-test-slave-q9j6 krb5kdc[8655](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.1@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
I20260620 14:48:31.771272 9679 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:48:31.762466 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:41725 (local address 127.0.97.1:47069)
0620 14:48:31.763062 (+ 596us) negotiation.cc:107] Waiting for socket to connect
0620 14:48:31.763080 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0620 14:48:31.763204 (+ 124us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 14:48:31.763509 (+ 305us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 14:48:31.763512 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 14:48:31.763987 (+ 475us) client_negotiation.cc:189] Negotiated authn=SASL
0620 14:48:31.764158 (+ 171us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:48:31.764167 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:48:31.765211 (+ 1044us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:48:31.765215 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:48:31.766232 (+ 1017us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:48:31.766245 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:48:31.766961 (+ 716us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:48:31.766966 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:48:31.767094 (+ 128us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:48:31.767875 (+ 781us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 14:48:31.767895 (+ 20us) client_negotiation.cc:656] calling sasl_client_start()
0620 14:48:31.770909 (+ 3014us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.97.62:41725: Server oryx/127.0.97.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":465,"thread_start_us":327,"threads_started":1}
W20260620 14:48:31.771384 9656 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 14:48:31.762616 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.1:47069 (local address 127.0.97.62:41725)
0620 14:48:31.762821 (+ 205us) server_negotiation.cc:206] Beginning negotiation
0620 14:48:31.762825 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 14:48:31.763162 (+ 337us) server_negotiation.cc:407] Connection header received
0620 14:48:31.763306 (+ 144us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:48:31.763310 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:48:31.763368 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:48:31.763459 (+ 91us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:48:31.764293 (+ 834us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:48:31.765062 (+ 769us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:48:31.766365 (+ 1303us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:48:31.766789 (+ 424us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:48:31.766890 (+ 101us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:48:31.771149 (+ 4259us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.97.1:47069: BlockingRecv error: recv got EOF from 127.0.97.1:47069 (error 108)
Metrics: {"server-negotiator.queue_time_us":80}
Jun 20 14:48:32 dist-test-slave-q9j6 krb5kdc[8655](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.2@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
Jun 20 14:48:32 dist-test-slave-q9j6 krb5kdc[8655](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.2@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
I20260620 14:48:32.233558 9680 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:48:32.224566 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:41725 (local address 127.0.97.2:32785)
0620 14:48:32.225099 (+ 533us) negotiation.cc:107] Waiting for socket to connect
0620 14:48:32.225116 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0620 14:48:32.225235 (+ 119us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 14:48:32.225564 (+ 329us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 14:48:32.225567 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 14:48:32.225942 (+ 375us) client_negotiation.cc:189] Negotiated authn=SASL
0620 14:48:32.226107 (+ 165us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:48:32.226115 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:48:32.227408 (+ 1293us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:48:32.227417 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:48:32.228920 (+ 1503us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:48:32.228933 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:48:32.229738 (+ 805us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:48:32.229744 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:48:32.229915 (+ 171us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:48:32.230508 (+ 593us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 14:48:32.230526 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0620 14:48:32.233313 (+ 2787us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.97.62:41725: Server oryx/127.0.97.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":410,"thread_start_us":288,"threads_started":1}
W20260620 14:48:32.233784 9656 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 14:48:32.224630 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.2:32785 (local address 127.0.97.62:41725)
0620 14:48:32.224797 (+ 167us) server_negotiation.cc:206] Beginning negotiation
0620 14:48:32.224802 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 14:48:32.225193 (+ 391us) server_negotiation.cc:407] Connection header received
0620 14:48:32.225335 (+ 142us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:48:32.225339 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:48:32.225413 (+ 74us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:48:32.225514 (+ 101us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:48:32.226238 (+ 724us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:48:32.227258 (+ 1020us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:48:32.229071 (+ 1813us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:48:32.229554 (+ 483us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:48:32.229701 (+ 147us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:48:32.233552 (+ 3851us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.97.2:32785: BlockingRecv error: recv got EOF from 127.0.97.2:32785 (error 108)
Metrics: {"server-negotiator.queue_time_us":55}
Jun 20 14:48:32 dist-test-slave-q9j6 krb5kdc[8655](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.3@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
Jun 20 14:48:32 dist-test-slave-q9j6 krb5kdc[8655](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.3@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
I20260620 14:48:32.707695 9681 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:48:32.698787 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:41725 (local address 127.0.97.3:53447)
0620 14:48:32.699350 (+ 563us) negotiation.cc:107] Waiting for socket to connect
0620 14:48:32.699366 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0620 14:48:32.699527 (+ 161us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 14:48:32.699932 (+ 405us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 14:48:32.699936 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 14:48:32.700375 (+ 439us) client_negotiation.cc:189] Negotiated authn=SASL
0620 14:48:32.700527 (+ 152us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:48:32.700535 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:48:32.701543 (+ 1008us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:48:32.701548 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:48:32.702462 (+ 914us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:48:32.702473 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:48:32.703232 (+ 759us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:48:32.703238 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:48:32.703406 (+ 168us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:48:32.704112 (+ 706us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 14:48:32.704131 (+ 19us) client_negotiation.cc:656] calling sasl_client_start()
0620 14:48:32.707298 (+ 3167us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.97.62:41725: Server oryx/127.0.97.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":462,"thread_start_us":317,"threads_started":1}
W20260620 14:48:32.707801 9656 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 14:48:32.698917 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.3:53447 (local address 127.0.97.62:41725)
0620 14:48:32.699166 (+ 249us) server_negotiation.cc:206] Beginning negotiation
0620 14:48:32.699173 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0620 14:48:32.699460 (+ 287us) server_negotiation.cc:407] Connection header received
0620 14:48:32.699683 (+ 223us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:48:32.699694 (+ 11us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:48:32.699754 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:48:32.699900 (+ 146us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:48:32.700683 (+ 783us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:48:32.701392 (+ 709us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:48:32.702608 (+ 1216us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:48:32.703037 (+ 429us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:48:32.703176 (+ 139us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:48:32.707543 (+ 4367us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.97.3:53447: BlockingRecv error: recv got EOF from 127.0.97.3:53447 (error 108)
Metrics: {"server-negotiator.queue_time_us":91}
Jun 20 14:48:32 dist-test-slave-q9j6 krb5kdc[8655](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.1@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
Jun 20 14:48:32 dist-test-slave-q9j6 krb5kdc[8655](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.1@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
I20260620 14:48:32.780781 9682 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:48:32.772357 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:41725 (local address 127.0.97.1:52751)
0620 14:48:32.772872 (+ 515us) negotiation.cc:107] Waiting for socket to connect
0620 14:48:32.772887 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0620 14:48:32.773049 (+ 162us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 14:48:32.773382 (+ 333us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 14:48:32.773387 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 14:48:32.773789 (+ 402us) client_negotiation.cc:189] Negotiated authn=SASL
0620 14:48:32.773950 (+ 161us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:48:32.773959 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:48:32.774964 (+ 1005us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:48:32.774971 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:48:32.775925 (+ 954us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:48:32.775936 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:48:32.776627 (+ 691us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:48:32.776633 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:48:32.776806 (+ 173us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:48:32.777374 (+ 568us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 14:48:32.777390 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0620 14:48:32.780466 (+ 3076us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.97.62:41725: Server oryx/127.0.97.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":428,"thread_start_us":288,"threads_started":1}
W20260620 14:48:32.780922 9656 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 14:48:32.772411 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.1:52751 (local address 127.0.97.62:41725)
0620 14:48:32.772657 (+ 246us) server_negotiation.cc:206] Beginning negotiation
0620 14:48:32.772666 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0620 14:48:32.772974 (+ 308us) server_negotiation.cc:407] Connection header received
0620 14:48:32.773173 (+ 199us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:48:32.773180 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:48:32.773244 (+ 64us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:48:32.773347 (+ 103us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:48:32.774097 (+ 750us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:48:32.774819 (+ 722us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:48:32.776077 (+ 1258us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:48:32.776475 (+ 398us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:48:32.776614 (+ 139us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:48:32.780625 (+ 4011us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.97.1:52751: BlockingRecv error: recv got EOF from 127.0.97.1:52751 (error 108)
Metrics: {"server-negotiator.queue_time_us":61}
Jun 20 14:48:33 dist-test-slave-q9j6 krb5kdc[8655](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.2@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
Jun 20 14:48:33 dist-test-slave-q9j6 krb5kdc[8655](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.2@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
I20260620 14:48:33.242184 9683 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:48:33.234528 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:41725 (local address 127.0.97.2:55563)
0620 14:48:33.235056 (+ 528us) negotiation.cc:107] Waiting for socket to connect
0620 14:48:33.235072 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0620 14:48:33.235182 (+ 110us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 14:48:33.235460 (+ 278us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 14:48:33.235463 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 14:48:33.235862 (+ 399us) client_negotiation.cc:189] Negotiated authn=SASL
0620 14:48:33.236032 (+ 170us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:48:33.236041 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:48:33.237026 (+ 985us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:48:33.237031 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:48:33.237956 (+ 925us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:48:33.237967 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:48:33.238615 (+ 648us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:48:33.238619 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:48:33.238736 (+ 117us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:48:33.239311 (+ 575us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 14:48:33.239326 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0620 14:48:33.241950 (+ 2624us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.97.62:41725: Server oryx/127.0.97.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":420,"thread_start_us":288,"threads_started":1}
W20260620 14:48:33.242352 9656 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 14:48:33.234555 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.2:55563 (local address 127.0.97.62:41725)
0620 14:48:33.234737 (+ 182us) server_negotiation.cc:206] Beginning negotiation
0620 14:48:33.234742 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 14:48:33.235142 (+ 400us) server_negotiation.cc:407] Connection header received
0620 14:48:33.235272 (+ 130us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:48:33.235276 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:48:33.235333 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:48:33.235416 (+ 83us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:48:33.236154 (+ 738us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:48:33.236907 (+ 753us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:48:33.238089 (+ 1182us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:48:33.238488 (+ 399us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:48:33.238580 (+ 92us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:48:33.242118 (+ 3538us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.97.2:55563: BlockingRecv error: recv got EOF from 127.0.97.2:55563 (error 108)
Metrics: {"server-negotiator.queue_time_us":69}
Jun 20 14:48:33 dist-test-slave-q9j6 krb5kdc[8655](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.3@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
Jun 20 14:48:33 dist-test-slave-q9j6 krb5kdc[8655](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.3@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
I20260620 14:48:33.717145 9684 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:48:33.708763 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:41725 (local address 127.0.97.3:43873)
0620 14:48:33.709364 (+ 601us) negotiation.cc:107] Waiting for socket to connect
0620 14:48:33.709385 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0620 14:48:33.709589 (+ 204us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 14:48:33.709951 (+ 362us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 14:48:33.709955 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 14:48:33.710376 (+ 421us) client_negotiation.cc:189] Negotiated authn=SASL
0620 14:48:33.710541 (+ 165us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:48:33.710549 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:48:33.711539 (+ 990us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:48:33.711545 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:48:33.712522 (+ 977us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:48:33.712534 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:48:33.713283 (+ 749us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:48:33.713289 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:48:33.713473 (+ 184us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:48:33.714053 (+ 580us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 14:48:33.714071 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0620 14:48:33.716870 (+ 2799us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.97.62:41725: Server oryx/127.0.97.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":464,"thread_start_us":306,"threads_started":1}
W20260620 14:48:33.717290 9656 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 14:48:33.708899 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.3:43873 (local address 127.0.97.62:41725)
0620 14:48:33.709129 (+ 230us) server_negotiation.cc:206] Beginning negotiation
0620 14:48:33.709136 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0620 14:48:33.709489 (+ 353us) server_negotiation.cc:407] Connection header received
0620 14:48:33.709731 (+ 242us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:48:33.709737 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:48:33.709813 (+ 76us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:48:33.709921 (+ 108us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:48:33.710696 (+ 775us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:48:33.711405 (+ 709us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:48:33.712678 (+ 1273us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:48:33.713112 (+ 434us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:48:33.713249 (+ 137us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:48:33.717085 (+ 3836us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.97.3:43873: BlockingRecv error: recv got EOF from 127.0.97.3:43873 (error 108)
Metrics: {"server-negotiator.queue_time_us":45}
Jun 20 14:48:33 dist-test-slave-q9j6 krb5kdc[8655](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.1@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
Jun 20 14:48:33 dist-test-slave-q9j6 krb5kdc[8655](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.1@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
I20260620 14:48:33.790426 9685 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:48:33.781911 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:41725 (local address 127.0.97.1:36125)
0620 14:48:33.782418 (+ 507us) negotiation.cc:107] Waiting for socket to connect
0620 14:48:33.782459 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0620 14:48:33.782592 (+ 133us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 14:48:33.783004 (+ 412us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 14:48:33.783009 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 14:48:33.783487 (+ 478us) client_negotiation.cc:189] Negotiated authn=SASL
0620 14:48:33.783717 (+ 230us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:48:33.783727 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:48:33.784834 (+ 1107us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:48:33.784838 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:48:33.785889 (+ 1051us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:48:33.785901 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:48:33.786594 (+ 693us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:48:33.786597 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:48:33.786717 (+ 120us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:48:33.787361 (+ 644us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 14:48:33.787379 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0620 14:48:33.790180 (+ 2801us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.97.62:41725: Server oryx/127.0.97.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":413,"thread_start_us":287,"threads_started":1}
W20260620 14:48:33.790699 9656 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 14:48:33.782005 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.1:36125 (local address 127.0.97.62:41725)
0620 14:48:33.782252 (+ 247us) server_negotiation.cc:206] Beginning negotiation
0620 14:48:33.782258 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 14:48:33.782547 (+ 289us) server_negotiation.cc:407] Connection header received
0620 14:48:33.782731 (+ 184us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:48:33.782736 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:48:33.782826 (+ 90us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:48:33.782932 (+ 106us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:48:33.783876 (+ 944us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:48:33.784705 (+ 829us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:48:33.786024 (+ 1319us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:48:33.786454 (+ 430us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:48:33.786560 (+ 106us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:48:33.790410 (+ 3850us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.97.1:36125: BlockingRecv error: recv got EOF from 127.0.97.1:36125 (error 108)
Metrics: {"server-negotiator.queue_time_us":61}
Jun 20 14:48:34 dist-test-slave-q9j6 krb5kdc[8655](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.2@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
Jun 20 14:48:34 dist-test-slave-q9j6 krb5kdc[8655](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.2@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
I20260620 14:48:34.252744 9686 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:48:34.243291 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:41725 (local address 127.0.97.2:49719)
0620 14:48:34.243867 (+ 576us) negotiation.cc:107] Waiting for socket to connect
0620 14:48:34.243891 (+ 24us) client_negotiation.cc:174] Beginning negotiation
0620 14:48:34.244084 (+ 193us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 14:48:34.244536 (+ 452us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 14:48:34.244543 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 14:48:34.245114 (+ 571us) client_negotiation.cc:189] Negotiated authn=SASL
0620 14:48:34.245295 (+ 181us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:48:34.245305 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:48:34.246334 (+ 1029us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:48:34.246339 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:48:34.247737 (+ 1398us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:48:34.247753 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:48:34.248599 (+ 846us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:48:34.248606 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:48:34.248788 (+ 182us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:48:34.249458 (+ 670us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 14:48:34.249479 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0620 14:48:34.252418 (+ 2939us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.97.62:41725: Server oryx/127.0.97.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":454,"thread_start_us":343,"threads_started":1}
W20260620 14:48:34.252869 9656 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 14:48:34.243364 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.2:49719 (local address 127.0.97.62:41725)
0620 14:48:34.243553 (+ 189us) server_negotiation.cc:206] Beginning negotiation
0620 14:48:34.243557 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 14:48:34.244041 (+ 484us) server_negotiation.cc:407] Connection header received
0620 14:48:34.244226 (+ 185us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:48:34.244232 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:48:34.244349 (+ 117us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:48:34.244454 (+ 105us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:48:34.245476 (+ 1022us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:48:34.246170 (+ 694us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:48:34.248003 (+ 1833us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:48:34.248409 (+ 406us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:48:34.248565 (+ 156us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:48:34.252628 (+ 4063us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.97.2:49719: BlockingRecv error: recv got EOF from 127.0.97.2:49719 (error 108)
Metrics: {"server-negotiator.queue_time_us":69}
Jun 20 14:48:34 dist-test-slave-q9j6 krb5kdc[8655](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.3@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
Jun 20 14:48:34 dist-test-slave-q9j6 krb5kdc[8655](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.3@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
I20260620 14:48:34.726701 9687 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:48:34.718088 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:41725 (local address 127.0.97.3:52211)
0620 14:48:34.718621 (+ 533us) negotiation.cc:107] Waiting for socket to connect
0620 14:48:34.718639 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0620 14:48:34.718754 (+ 115us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 14:48:34.719059 (+ 305us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 14:48:34.719063 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 14:48:34.719410 (+ 347us) client_negotiation.cc:189] Negotiated authn=SASL
0620 14:48:34.719549 (+ 139us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:48:34.719557 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:48:34.720664 (+ 1107us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:48:34.720676 (+ 12us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:48:34.721586 (+ 910us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:48:34.721598 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:48:34.722395 (+ 797us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:48:34.722400 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:48:34.722580 (+ 180us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:48:34.723314 (+ 734us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 14:48:34.723357 (+ 43us) client_negotiation.cc:656] calling sasl_client_start()
0620 14:48:34.726468 (+ 3111us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.97.62:41725: Server oryx/127.0.97.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":412,"thread_start_us":314,"threads_started":1}
W20260620 14:48:34.726948 9656 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 14:48:34.718310 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.3:52211 (local address 127.0.97.62:41725)
0620 14:48:34.718527 (+ 217us) server_negotiation.cc:206] Beginning negotiation
0620 14:48:34.718532 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 14:48:34.718719 (+ 187us) server_negotiation.cc:407] Connection header received
0620 14:48:34.718854 (+ 135us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:48:34.718858 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:48:34.718915 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:48:34.719007 (+ 92us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:48:34.719745 (+ 738us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:48:34.720497 (+ 752us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:48:34.721758 (+ 1261us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:48:34.722208 (+ 450us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:48:34.722340 (+ 132us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:48:34.726684 (+ 4344us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.97.3:52211: BlockingRecv error: recv got EOF from 127.0.97.3:52211 (error 108)
Metrics: {"server-negotiator.queue_time_us":105}
Jun 20 14:48:34 dist-test-slave-q9j6 krb5kdc[8655](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.1@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
Jun 20 14:48:34 dist-test-slave-q9j6 krb5kdc[8655](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.1@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
I20260620 14:48:34.799949 9688 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:48:34.791461 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:41725 (local address 127.0.97.1:46857)
0620 14:48:34.791965 (+ 504us) negotiation.cc:107] Waiting for socket to connect
0620 14:48:34.791989 (+ 24us) client_negotiation.cc:174] Beginning negotiation
0620 14:48:34.792102 (+ 113us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 14:48:34.792391 (+ 289us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 14:48:34.792394 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 14:48:34.792779 (+ 385us) client_negotiation.cc:189] Negotiated authn=SASL
0620 14:48:34.792959 (+ 180us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:48:34.792978 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:48:34.793944 (+ 966us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:48:34.793948 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:48:34.794843 (+ 895us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:48:34.794855 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:48:34.795754 (+ 899us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:48:34.795758 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:48:34.795883 (+ 125us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:48:34.796619 (+ 736us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 14:48:34.796645 (+ 26us) client_negotiation.cc:656] calling sasl_client_start()
0620 14:48:34.799694 (+ 3049us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.97.62:41725: Server oryx/127.0.97.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":412,"thread_start_us":304,"threads_started":1}
W20260620 14:48:34.800082 9656 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 14:48:34.791639 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.1:46857 (local address 127.0.97.62:41725)
0620 14:48:34.791872 (+ 233us) server_negotiation.cc:206] Beginning negotiation
0620 14:48:34.791879 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0620 14:48:34.792068 (+ 189us) server_negotiation.cc:407] Connection header received
0620 14:48:34.792200 (+ 132us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:48:34.792203 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:48:34.792264 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:48:34.792348 (+ 84us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:48:34.793091 (+ 743us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:48:34.793814 (+ 723us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:48:34.795000 (+ 1186us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:48:34.795538 (+ 538us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:48:34.795699 (+ 161us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:48:34.799881 (+ 4182us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.97.1:46857: BlockingRecv error: recv got EOF from 127.0.97.1:46857 (error 108)
Metrics: {"server-negotiator.queue_time_us":66}
Jun 20 14:48:35 dist-test-slave-q9j6 krb5kdc[8655](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.2@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
Jun 20 14:48:35 dist-test-slave-q9j6 krb5kdc[8655](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.2@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
I20260620 14:48:35.261988 9689 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:48:35.253650 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:41725 (local address 127.0.97.2:57959)
0620 14:48:35.254163 (+ 513us) negotiation.cc:107] Waiting for socket to connect
0620 14:48:35.254179 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0620 14:48:35.254289 (+ 110us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 14:48:35.254580 (+ 291us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 14:48:35.254583 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 14:48:35.254961 (+ 378us) client_negotiation.cc:189] Negotiated authn=SASL
0620 14:48:35.255133 (+ 172us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:48:35.255141 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:48:35.256130 (+ 989us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:48:35.256135 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:48:35.257038 (+ 903us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:48:35.257050 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:48:35.257707 (+ 657us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:48:35.257711 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:48:35.257832 (+ 121us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:48:35.258529 (+ 697us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 14:48:35.258567 (+ 38us) client_negotiation.cc:656] calling sasl_client_start()
0620 14:48:35.261644 (+ 3077us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.97.62:41725: Server oryx/127.0.97.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":407,"thread_start_us":286,"threads_started":1}
W20260620 14:48:35.262125 9656 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 14:48:35.253790 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.2:57959 (local address 127.0.97.62:41725)
0620 14:48:35.253960 (+ 170us) server_negotiation.cc:206] Beginning negotiation
0620 14:48:35.253965 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 14:48:35.254251 (+ 286us) server_negotiation.cc:407] Connection header received
0620 14:48:35.254381 (+ 130us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:48:35.254384 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:48:35.254441 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:48:35.254531 (+ 90us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:48:35.255252 (+ 721us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:48:35.256006 (+ 754us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:48:35.257159 (+ 1153us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:48:35.257560 (+ 401us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:48:35.257648 (+ 88us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:48:35.261846 (+ 4198us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.97.2:57959: BlockingRecv error: recv got EOF from 127.0.97.2:57959 (error 108)
Metrics: {"server-negotiator.queue_time_us":60}
Jun 20 14:48:35 dist-test-slave-q9j6 krb5kdc[8655](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.3@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
Jun 20 14:48:35 dist-test-slave-q9j6 krb5kdc[8655](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.3@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
I20260620 14:48:35.738799 9690 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:48:35.727696 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:41725 (local address 127.0.97.3:35139)
0620 14:48:35.728325 (+ 629us) negotiation.cc:107] Waiting for socket to connect
0620 14:48:35.728341 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0620 14:48:35.728489 (+ 148us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 14:48:35.728866 (+ 377us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 14:48:35.728869 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 14:48:35.729390 (+ 521us) client_negotiation.cc:189] Negotiated authn=SASL
0620 14:48:35.729628 (+ 238us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:48:35.729640 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:48:35.730693 (+ 1053us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:48:35.730697 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:48:35.731586 (+ 889us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:48:35.731630 (+ 44us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:48:35.732252 (+ 622us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:48:35.732256 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:48:35.732370 (+ 114us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:48:35.735747 (+ 3377us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 14:48:35.735766 (+ 19us) client_negotiation.cc:656] calling sasl_client_start()
0620 14:48:35.738554 (+ 2788us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.97.62:41725: Server oryx/127.0.97.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":460,"thread_start_us":343,"threads_started":1}
W20260620 14:48:35.739089 9656 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 14:48:35.727787 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.3:35139 (local address 127.0.97.62:41725)
0620 14:48:35.727954 (+ 167us) server_negotiation.cc:206] Beginning negotiation
0620 14:48:35.727959 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 14:48:35.728437 (+ 478us) server_negotiation.cc:407] Connection header received
0620 14:48:35.728633 (+ 196us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:48:35.728637 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:48:35.728699 (+ 62us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:48:35.728833 (+ 134us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:48:35.729775 (+ 942us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:48:35.730575 (+ 800us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:48:35.731725 (+ 1150us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:48:35.732126 (+ 401us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:48:35.732217 (+ 91us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:48:35.738815 (+ 6598us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.97.3:35139: BlockingRecv error: recv got EOF from 127.0.97.3:35139 (error 108)
Metrics: {"server-negotiator.queue_time_us":57}
Jun 20 14:48:35 dist-test-slave-q9j6 krb5kdc[8655](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.1@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
Jun 20 14:48:35 dist-test-slave-q9j6 krb5kdc[8655](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.1@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
I20260620 14:48:35.812657 9691 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:48:35.801002 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:41725 (local address 127.0.97.1:54223)
0620 14:48:35.801870 (+ 868us) negotiation.cc:107] Waiting for socket to connect
0620 14:48:35.801894 (+ 24us) client_negotiation.cc:174] Beginning negotiation
0620 14:48:35.802076 (+ 182us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 14:48:35.802864 (+ 788us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 14:48:35.802868 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 14:48:35.803426 (+ 558us) client_negotiation.cc:189] Negotiated authn=SASL
0620 14:48:35.803673 (+ 247us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:48:35.803688 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:48:35.805311 (+ 1623us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:48:35.805318 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:48:35.806418 (+ 1100us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:48:35.806433 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:48:35.807516 (+ 1083us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:48:35.807526 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:48:35.807754 (+ 228us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:48:35.808604 (+ 850us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 14:48:35.808634 (+ 30us) client_negotiation.cc:656] calling sasl_client_start()
0620 14:48:35.812421 (+ 3787us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.97.62:41725: Server oryx/127.0.97.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":748,"thread_start_us":595,"threads_started":1}
W20260620 14:48:35.813026 9656 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 14:48:35.801215 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.1:54223 (local address 127.0.97.62:41725)
0620 14:48:35.801450 (+ 235us) server_negotiation.cc:206] Beginning negotiation
0620 14:48:35.801458 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0620 14:48:35.802003 (+ 545us) server_negotiation.cc:407] Connection header received
0620 14:48:35.802261 (+ 258us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:48:35.802269 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:48:35.802455 (+ 186us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:48:35.802583 (+ 128us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:48:35.803973 (+ 1390us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:48:35.805170 (+ 1197us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:48:35.806720 (+ 1550us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:48:35.807357 (+ 637us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:48:35.807455 (+ 98us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:48:35.812731 (+ 5276us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.97.1:54223: BlockingRecv error: recv got EOF from 127.0.97.1:54223 (error 108)
Metrics: {"server-negotiator.queue_time_us":72}
Jun 20 14:48:36 dist-test-slave-q9j6 krb5kdc[8655](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.2@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
Jun 20 14:48:36 dist-test-slave-q9j6 krb5kdc[8655](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.2@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
I20260620 14:48:36.271737 9692 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:48:36.262964 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:41725 (local address 127.0.97.2:39439)
0620 14:48:36.263533 (+ 569us) negotiation.cc:107] Waiting for socket to connect
0620 14:48:36.263550 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0620 14:48:36.263734 (+ 184us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 14:48:36.264055 (+ 321us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 14:48:36.264058 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 14:48:36.264420 (+ 362us) client_negotiation.cc:189] Negotiated authn=SASL
0620 14:48:36.264573 (+ 153us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:48:36.264580 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:48:36.265605 (+ 1025us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:48:36.265610 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:48:36.266587 (+ 977us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:48:36.266601 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:48:36.267259 (+ 658us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:48:36.267263 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:48:36.267399 (+ 136us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:48:36.268148 (+ 749us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 14:48:36.268169 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0620 14:48:36.271442 (+ 3273us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.97.62:41725: Server oryx/127.0.97.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":455,"thread_start_us":319,"threads_started":1}
W20260620 14:48:36.271934 9656 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 14:48:36.263069 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.2:39439 (local address 127.0.97.62:41725)
0620 14:48:36.263239 (+ 170us) server_negotiation.cc:206] Beginning negotiation
0620 14:48:36.263244 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 14:48:36.263649 (+ 405us) server_negotiation.cc:407] Connection header received
0620 14:48:36.263845 (+ 196us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:48:36.263849 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:48:36.263910 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:48:36.264000 (+ 90us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:48:36.264701 (+ 701us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:48:36.265437 (+ 736us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:48:36.266729 (+ 1292us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:48:36.267128 (+ 399us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:48:36.267232 (+ 104us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:48:36.271697 (+ 4465us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.97.2:39439: BlockingRecv error: recv got EOF from 127.0.97.2:39439 (error 108)
Metrics: {"server-negotiator.queue_time_us":59}
Jun 20 14:48:36 dist-test-slave-q9j6 krb5kdc[8655](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.3@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
Jun 20 14:48:36 dist-test-slave-q9j6 krb5kdc[8655](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.3@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
I20260620 14:48:36.748500 9693 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:48:36.739876 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:41725 (local address 127.0.97.3:33409)
0620 14:48:36.740410 (+ 534us) negotiation.cc:107] Waiting for socket to connect
0620 14:48:36.740426 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0620 14:48:36.740544 (+ 118us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 14:48:36.740856 (+ 312us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 14:48:36.740859 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 14:48:36.741243 (+ 384us) client_negotiation.cc:189] Negotiated authn=SASL
0620 14:48:36.741400 (+ 157us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:48:36.741407 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:48:36.742643 (+ 1236us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:48:36.742649 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:48:36.743782 (+ 1133us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:48:36.743795 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:48:36.744491 (+ 696us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:48:36.744497 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:48:36.744679 (+ 182us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:48:36.745294 (+ 615us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 14:48:36.745311 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0620 14:48:36.748150 (+ 2839us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.97.62:41725: Server oryx/127.0.97.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":423,"thread_start_us":295,"threads_started":1}
W20260620 14:48:36.748631 9656 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 14:48:36.739923 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.3:33409 (local address 127.0.97.62:41725)
0620 14:48:36.740097 (+ 174us) server_negotiation.cc:206] Beginning negotiation
0620 14:48:36.740102 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 14:48:36.740505 (+ 403us) server_negotiation.cc:407] Connection header received
0620 14:48:36.740646 (+ 141us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:48:36.740650 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:48:36.740710 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:48:36.740805 (+ 95us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:48:36.741593 (+ 788us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:48:36.742475 (+ 882us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:48:36.743927 (+ 1452us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:48:36.744332 (+ 405us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:48:36.744466 (+ 134us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:48:36.748370 (+ 3904us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.97.3:33409: BlockingRecv error: recv got EOF from 127.0.97.3:33409 (error 108)
Metrics: {"server-negotiator.queue_time_us":59}
Jun 20 14:48:36 dist-test-slave-q9j6 krb5kdc[8655](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.1@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
Jun 20 14:48:36 dist-test-slave-q9j6 krb5kdc[8655](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.1@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
I20260620 14:48:36.822746 9694 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:48:36.813818 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:41725 (local address 127.0.97.1:52251)
0620 14:48:36.814380 (+ 562us) negotiation.cc:107] Waiting for socket to connect
0620 14:48:36.814398 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0620 14:48:36.814578 (+ 180us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 14:48:36.814957 (+ 379us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 14:48:36.814961 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 14:48:36.815366 (+ 405us) client_negotiation.cc:189] Negotiated authn=SASL
0620 14:48:36.815586 (+ 220us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:48:36.815600 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:48:36.816938 (+ 1338us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:48:36.816944 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:48:36.817990 (+ 1046us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:48:36.818002 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:48:36.818751 (+ 749us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:48:36.818757 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:48:36.818926 (+ 169us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:48:36.819494 (+ 568us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 14:48:36.819512 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0620 14:48:36.822410 (+ 2898us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.97.62:41725: Server oryx/127.0.97.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":458,"thread_start_us":279,"threads_started":1}
W20260620 14:48:36.822938 9656 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 14:48:36.813943 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.1:52251 (local address 127.0.97.62:41725)
0620 14:48:36.814217 (+ 274us) server_negotiation.cc:206] Beginning negotiation
0620 14:48:36.814224 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0620 14:48:36.814504 (+ 280us) server_negotiation.cc:407] Connection header received
0620 14:48:36.814714 (+ 210us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:48:36.814719 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:48:36.814787 (+ 68us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:48:36.814898 (+ 111us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:48:36.815764 (+ 866us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:48:36.816749 (+ 985us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:48:36.818141 (+ 1392us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:48:36.818570 (+ 429us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:48:36.818712 (+ 142us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:48:36.822652 (+ 3940us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.97.1:52251: BlockingRecv error: recv got EOF from 127.0.97.1:52251 (error 108)
Metrics: {"server-negotiator.queue_time_us":78}
Jun 20 14:48:37 dist-test-slave-q9j6 krb5kdc[8655](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.2@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
Jun 20 14:48:37 dist-test-slave-q9j6 krb5kdc[8655](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.2@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
I20260620 14:48:37.281606 9695 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:48:37.272835 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:41725 (local address 127.0.97.2:46355)
0620 14:48:37.273447 (+ 612us) negotiation.cc:107] Waiting for socket to connect
0620 14:48:37.273470 (+ 23us) client_negotiation.cc:174] Beginning negotiation
0620 14:48:37.273637 (+ 167us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 14:48:37.273968 (+ 331us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 14:48:37.273973 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 14:48:37.274415 (+ 442us) client_negotiation.cc:189] Negotiated authn=SASL
0620 14:48:37.274599 (+ 184us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:48:37.274609 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:48:37.275704 (+ 1095us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:48:37.275710 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:48:37.276658 (+ 948us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:48:37.276671 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:48:37.277462 (+ 791us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:48:37.277469 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:48:37.277649 (+ 180us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:48:37.278233 (+ 584us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 14:48:37.278250 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0620 14:48:37.281245 (+ 2995us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.97.62:41725: Server oryx/127.0.97.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":472,"thread_start_us":277,"threads_started":1}
W20260620 14:48:37.281697 9656 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 14:48:37.273009 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.2:46355 (local address 127.0.97.62:41725)
0620 14:48:37.273220 (+ 211us) server_negotiation.cc:206] Beginning negotiation
0620 14:48:37.273226 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 14:48:37.273589 (+ 363us) server_negotiation.cc:407] Connection header received
0620 14:48:37.273762 (+ 173us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:48:37.273775 (+ 13us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:48:37.273833 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:48:37.273938 (+ 105us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:48:37.274755 (+ 817us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:48:37.275523 (+ 768us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:48:37.276810 (+ 1287us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:48:37.277271 (+ 461us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:48:37.277416 (+ 145us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:48:37.281457 (+ 4041us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.97.2:46355: BlockingRecv error: recv got EOF from 127.0.97.2:46355 (error 108)
Metrics: {"server-negotiator.queue_time_us":79}
Jun 20 14:48:37 dist-test-slave-q9j6 krb5kdc[8655](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.3@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
Jun 20 14:48:37 dist-test-slave-q9j6 krb5kdc[8655](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.3@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
I20260620 14:48:37.757892 9696 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:48:37.749618 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:41725 (local address 127.0.97.3:53545)
0620 14:48:37.750101 (+ 483us) negotiation.cc:107] Waiting for socket to connect
0620 14:48:37.750140 (+ 39us) client_negotiation.cc:174] Beginning negotiation
0620 14:48:37.750316 (+ 176us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 14:48:37.750666 (+ 350us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 14:48:37.750671 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 14:48:37.751093 (+ 422us) client_negotiation.cc:189] Negotiated authn=SASL
0620 14:48:37.751245 (+ 152us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:48:37.751253 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:48:37.752366 (+ 1113us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:48:37.752372 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:48:37.753321 (+ 949us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:48:37.753333 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:48:37.754086 (+ 753us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:48:37.754094 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:48:37.754258 (+ 164us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:48:37.754880 (+ 622us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 14:48:37.754898 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0620 14:48:37.757660 (+ 2762us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.97.62:41725: Server oryx/127.0.97.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":365,"thread_start_us":351,"threads_started":1}
W20260620 14:48:37.758116 9656 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 14:48:37.749773 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.3:53545 (local address 127.0.97.62:41725)
0620 14:48:37.749960 (+ 187us) server_negotiation.cc:206] Beginning negotiation
0620 14:48:37.749966 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 14:48:37.750247 (+ 281us) server_negotiation.cc:407] Connection header received
0620 14:48:37.750444 (+ 197us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:48:37.750450 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:48:37.750527 (+ 77us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:48:37.750632 (+ 105us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:48:37.751392 (+ 760us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:48:37.752220 (+ 828us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:48:37.753492 (+ 1272us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:48:37.753888 (+ 396us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:48:37.754021 (+ 133us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:48:37.757871 (+ 3850us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.97.3:53545: BlockingRecv error: recv got EOF from 127.0.97.3:53545 (error 108)
Metrics: {"server-negotiator.queue_time_us":42}
Jun 20 14:48:37 dist-test-slave-q9j6 krb5kdc[8655](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.1@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
Jun 20 14:48:37 dist-test-slave-q9j6 krb5kdc[8655](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.1@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
I20260620 14:48:37.832691 9697 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:48:37.823798 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:41725 (local address 127.0.97.1:59803)
0620 14:48:37.824324 (+ 526us) negotiation.cc:107] Waiting for socket to connect
0620 14:48:37.824343 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0620 14:48:37.824467 (+ 124us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 14:48:37.824784 (+ 317us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 14:48:37.824786 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 14:48:37.825129 (+ 343us) client_negotiation.cc:189] Negotiated authn=SASL
0620 14:48:37.825296 (+ 167us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:48:37.825304 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:48:37.826335 (+ 1031us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:48:37.826339 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:48:37.827255 (+ 916us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:48:37.827265 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:48:37.828435 (+ 1170us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:48:37.828440 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:48:37.828604 (+ 164us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:48:37.829263 (+ 659us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 14:48:37.829280 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0620 14:48:37.832313 (+ 3033us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.97.62:41725: Server oryx/127.0.97.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":379,"thread_start_us":274,"threads_started":1}
W20260620 14:48:37.832805 9656 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 14:48:37.823860 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.1:59803 (local address 127.0.97.62:41725)
0620 14:48:37.824116 (+ 256us) server_negotiation.cc:206] Beginning negotiation
0620 14:48:37.824121 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 14:48:37.824426 (+ 305us) server_negotiation.cc:407] Connection header received
0620 14:48:37.824587 (+ 161us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:48:37.824591 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:48:37.824655 (+ 64us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:48:37.824749 (+ 94us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:48:37.825423 (+ 674us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:48:37.826200 (+ 777us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:48:37.827385 (+ 1185us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:48:37.828224 (+ 839us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:48:37.828367 (+ 143us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:48:37.832534 (+ 4167us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.97.1:59803: BlockingRecv error: recv got EOF from 127.0.97.1:59803 (error 108)
Metrics: {"server-negotiator.queue_time_us":132}
Jun 20 14:48:38 dist-test-slave-q9j6 krb5kdc[8655](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.2@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
Jun 20 14:48:38 dist-test-slave-q9j6 krb5kdc[8655](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.2@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
I20260620 14:48:38.291122 9698 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:48:38.282547 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:41725 (local address 127.0.97.2:41235)
0620 14:48:38.283021 (+ 474us) negotiation.cc:107] Waiting for socket to connect
0620 14:48:38.283037 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0620 14:48:38.283208 (+ 171us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 14:48:38.283565 (+ 357us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 14:48:38.283570 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 14:48:38.283985 (+ 415us) client_negotiation.cc:189] Negotiated authn=SASL
0620 14:48:38.284159 (+ 174us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:48:38.284168 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:48:38.285255 (+ 1087us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:48:38.285261 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:48:38.286260 (+ 999us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:48:38.286272 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:48:38.287043 (+ 771us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:48:38.287049 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:48:38.287217 (+ 168us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:48:38.287959 (+ 742us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 14:48:38.287977 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0620 14:48:38.290761 (+ 2784us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.97.62:41725: Server oryx/127.0.97.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":376,"thread_start_us":239,"threads_started":1}
W20260620 14:48:38.291199 9656 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 14:48:38.282642 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.2:41235 (local address 127.0.97.62:41725)
0620 14:48:38.282908 (+ 266us) server_negotiation.cc:206] Beginning negotiation
0620 14:48:38.282915 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0620 14:48:38.283117 (+ 202us) server_negotiation.cc:407] Connection header received
0620 14:48:38.283336 (+ 219us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:48:38.283341 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:48:38.283428 (+ 87us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:48:38.283530 (+ 102us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:48:38.284318 (+ 788us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:48:38.285117 (+ 799us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:48:38.286436 (+ 1319us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:48:38.286885 (+ 449us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:48:38.287017 (+ 132us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:48:38.290961 (+ 3944us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.97.2:41235: BlockingRecv error: recv got EOF from 127.0.97.2:41235 (error 108)
Metrics: {"server-negotiator.queue_time_us":69}
Jun 20 14:48:38 dist-test-slave-q9j6 krb5kdc[8655](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.3@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
Jun 20 14:48:38 dist-test-slave-q9j6 krb5kdc[8655](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.3@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
I20260620 14:48:38.767391 9699 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:48:38.758820 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:41725 (local address 127.0.97.3:33831)
0620 14:48:38.759391 (+ 571us) negotiation.cc:107] Waiting for socket to connect
0620 14:48:38.759415 (+ 24us) client_negotiation.cc:174] Beginning negotiation
0620 14:48:38.759626 (+ 211us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 14:48:38.759999 (+ 373us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 14:48:38.760004 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 14:48:38.760409 (+ 405us) client_negotiation.cc:189] Negotiated authn=SASL
0620 14:48:38.760576 (+ 167us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:48:38.760586 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:48:38.761616 (+ 1030us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:48:38.761622 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:48:38.762551 (+ 929us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:48:38.762562 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:48:38.763322 (+ 760us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:48:38.763327 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:48:38.763516 (+ 189us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:48:38.764259 (+ 743us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 14:48:38.764276 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0620 14:48:38.767035 (+ 2759us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.97.62:41725: Server oryx/127.0.97.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":403,"thread_start_us":236,"threads_started":1}
W20260620 14:48:38.767520 9656 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 14:48:38.758900 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.3:33831 (local address 127.0.97.62:41725)
0620 14:48:38.759079 (+ 179us) server_negotiation.cc:206] Beginning negotiation
0620 14:48:38.759089 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0620 14:48:38.759506 (+ 417us) server_negotiation.cc:407] Connection header received
0620 14:48:38.759750 (+ 244us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:48:38.759755 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:48:38.759826 (+ 71us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:48:38.759976 (+ 150us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:48:38.760724 (+ 748us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:48:38.761461 (+ 737us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:48:38.762693 (+ 1232us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:48:38.763165 (+ 472us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:48:38.763299 (+ 134us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:48:38.767274 (+ 3975us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.97.3:33831: BlockingRecv error: recv got EOF from 127.0.97.3:33831 (error 108)
Metrics: {"server-negotiator.queue_time_us":41}
Jun 20 14:48:38 dist-test-slave-q9j6 krb5kdc[8655](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.1@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
Jun 20 14:48:38 dist-test-slave-q9j6 krb5kdc[8655](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.1@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
I20260620 14:48:38.841579 9700 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:48:38.833826 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:41725 (local address 127.0.97.1:46831)
0620 14:48:38.834282 (+ 456us) negotiation.cc:107] Waiting for socket to connect
0620 14:48:38.834298 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0620 14:48:38.834400 (+ 102us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 14:48:38.834723 (+ 323us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 14:48:38.834727 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 14:48:38.835149 (+ 422us) client_negotiation.cc:189] Negotiated authn=SASL
0620 14:48:38.835347 (+ 198us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:48:38.835357 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:48:38.836317 (+ 960us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:48:38.836321 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:48:38.837224 (+ 903us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:48:38.837235 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:48:38.837881 (+ 646us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:48:38.837885 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:48:38.838014 (+ 129us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:48:38.838551 (+ 537us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 14:48:38.838568 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0620 14:48:38.841318 (+ 2750us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.97.62:41725: Server oryx/127.0.97.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":360,"thread_start_us":269,"threads_started":1}
W20260620 14:48:38.841740 9656 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 14:48:38.833908 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.1:46831 (local address 127.0.97.62:41725)
0620 14:48:38.834138 (+ 230us) server_negotiation.cc:206] Beginning negotiation
0620 14:48:38.834144 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 14:48:38.834374 (+ 230us) server_negotiation.cc:407] Connection header received
0620 14:48:38.834481 (+ 107us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:48:38.834484 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:48:38.834555 (+ 71us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:48:38.834636 (+ 81us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:48:38.835479 (+ 843us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:48:38.836197 (+ 718us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:48:38.837346 (+ 1149us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:48:38.837763 (+ 417us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:48:38.837853 (+ 90us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:48:38.841513 (+ 3660us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.97.1:46831: BlockingRecv error: recv got EOF from 127.0.97.1:46831 (error 108)
Metrics: {"server-negotiator.queue_time_us":73}
Jun 20 14:48:39 dist-test-slave-q9j6 krb5kdc[8655](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.2@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
Jun 20 14:48:39 dist-test-slave-q9j6 krb5kdc[8655](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.2@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
I20260620 14:48:39.301155 9701 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:48:39.292452 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:41725 (local address 127.0.97.2:40871)
0620 14:48:39.293123 (+ 671us) negotiation.cc:107] Waiting for socket to connect
0620 14:48:39.293140 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0620 14:48:39.293303 (+ 163us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 14:48:39.293654 (+ 351us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 14:48:39.293659 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 14:48:39.294112 (+ 453us) client_negotiation.cc:189] Negotiated authn=SASL
0620 14:48:39.294293 (+ 181us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:48:39.294302 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:48:39.295328 (+ 1026us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:48:39.295334 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:48:39.296546 (+ 1212us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:48:39.296558 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:48:39.297289 (+ 731us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:48:39.297295 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:48:39.297458 (+ 163us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:48:39.298047 (+ 589us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 14:48:39.298064 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0620 14:48:39.300782 (+ 2718us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.97.62:41725: Server oryx/127.0.97.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":539,"thread_start_us":409,"threads_started":1}
W20260620 14:48:39.301234 9656 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 14:48:39.292609 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.2:40871 (local address 127.0.97.62:41725)
0620 14:48:39.292939 (+ 330us) server_negotiation.cc:206] Beginning negotiation
0620 14:48:39.292952 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0620 14:48:39.293234 (+ 282us) server_negotiation.cc:407] Connection header received
0620 14:48:39.293432 (+ 198us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:48:39.293441 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:48:39.293520 (+ 79us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:48:39.293625 (+ 105us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:48:39.294439 (+ 814us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:48:39.295188 (+ 749us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:48:39.296676 (+ 1488us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:48:39.297138 (+ 462us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:48:39.297270 (+ 132us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:48:39.300991 (+ 3721us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.97.2:40871: BlockingRecv error: recv got EOF from 127.0.97.2:40871 (error 108)
Metrics: {"server-negotiator.queue_time_us":169}
I20260620 14:48:39.588402 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskUci3DU/build/asan/bin/kudu pid 9243
I20260620 14:48:39.600360 9703 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:48:39.589477 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56610 (local address 127.0.97.1:33589)
0620 14:48:39.590091 (+ 614us) server_negotiation.cc:206] Beginning negotiation
0620 14:48:39.590099 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0620 14:48:39.590125 (+ 26us) server_negotiation.cc:407] Connection header received
0620 14:48:39.590186 (+ 61us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:48:39.590192 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:48:39.590417 (+ 225us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:48:39.590583 (+ 166us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:48:39.591954 (+ 1371us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:48:39.592991 (+ 1037us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:48:39.594247 (+ 1256us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:48:39.594919 (+ 672us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:48:39.595021 (+ 102us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:48:39.597444 (+ 2423us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 14:48:39.597453 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 14:48:39.597468 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 14:48:39.597504 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0620 14:48:39.599861 (+ 2357us) server_negotiation.cc:394] Sending RPC error: FATAL_UNAUTHORIZED: Not authorized: Request ticket server kudu/127.0.97.1@KRBTEST.COM not found in keytab (ticket kvno 2)
0620 14:48:39.600134 (+ 273us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:56610: Request ticket server kudu/127.0.97.1@KRBTEST.COM not found in keytab (ticket kvno 2)
Metrics: {"server-negotiator.queue_time_us":448,"thread_start_us":329,"threads_started":1}
W20260620 14:48:39.600481 9703 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:56610: Request ticket server kudu/127.0.97.1@KRBTEST.COM not found in keytab (ticket kvno 2)
W20260620 14:48:39.600549 388 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskUci3DU/build/asan/bin/kudu pid 9243: Not authorized: Client connection negotiation failed: client connection to 127.0.97.1:33589: FATAL_UNAUTHORIZED: Not authorized: Request ticket server kudu/127.0.97.1@KRBTEST.COM not found in keytab (ticket kvno 2)
I20260620 14:48:39.600771 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskUci3DU/build/asan/bin/kudu with pid 9243
I20260620 14:48:39.617697 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskUci3DU/build/asan/bin/kudu pid 9394
I20260620 14:48:39.630052 9704 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:48:39.618816 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58520 (local address 127.0.97.2:40895)
0620 14:48:39.619457 (+ 641us) server_negotiation.cc:206] Beginning negotiation
0620 14:48:39.619465 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0620 14:48:39.619496 (+ 31us) server_negotiation.cc:407] Connection header received
0620 14:48:39.619560 (+ 64us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:48:39.619566 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:48:39.619880 (+ 314us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:48:39.620072 (+ 192us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:48:39.621269 (+ 1197us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:48:39.622294 (+ 1025us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:48:39.623725 (+ 1431us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:48:39.624358 (+ 633us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:48:39.624455 (+ 97us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:48:39.626827 (+ 2372us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 14:48:39.626835 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 14:48:39.626850 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 14:48:39.626885 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0620 14:48:39.629520 (+ 2635us) server_negotiation.cc:394] Sending RPC error: FATAL_UNAUTHORIZED: Not authorized: Request ticket server kudu/127.0.97.2@KRBTEST.COM not found in keytab (ticket kvno 2)
0620 14:48:39.629776 (+ 256us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:58520: Request ticket server kudu/127.0.97.2@KRBTEST.COM not found in keytab (ticket kvno 2)
Metrics: {"server-negotiator.queue_time_us":499,"thread_start_us":344,"threads_started":1}
W20260620 14:48:39.630160 9704 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:58520: Request ticket server kudu/127.0.97.2@KRBTEST.COM not found in keytab (ticket kvno 2)
W20260620 14:48:39.630170 388 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskUci3DU/build/asan/bin/kudu pid 9394: Not authorized: Client connection negotiation failed: client connection to 127.0.97.2:40895: FATAL_UNAUTHORIZED: Not authorized: Request ticket server kudu/127.0.97.2@KRBTEST.COM not found in keytab (ticket kvno 2)
I20260620 14:48:39.630342 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskUci3DU/build/asan/bin/kudu with pid 9394
I20260620 14:48:39.646714 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskUci3DU/build/asan/bin/kudu pid 9525
I20260620 14:48:39.658804 9705 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:48:39.647705 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39152 (local address 127.0.97.3:34719)
0620 14:48:39.648291 (+ 586us) server_negotiation.cc:206] Beginning negotiation
0620 14:48:39.648299 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0620 14:48:39.648325 (+ 26us) server_negotiation.cc:407] Connection header received
0620 14:48:39.648384 (+ 59us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:48:39.648392 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:48:39.648626 (+ 234us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:48:39.648780 (+ 154us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:48:39.650087 (+ 1307us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:48:39.651196 (+ 1109us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:48:39.652529 (+ 1333us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:48:39.653077 (+ 548us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:48:39.653179 (+ 102us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:48:39.655580 (+ 2401us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 14:48:39.655589 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 14:48:39.655649 (+ 60us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 14:48:39.655714 (+ 65us) server_negotiation.cc:922] calling sasl_server_start()
0620 14:48:39.658266 (+ 2552us) server_negotiation.cc:394] Sending RPC error: FATAL_UNAUTHORIZED: Not authorized: Request ticket server kudu/127.0.97.3@KRBTEST.COM not found in keytab (ticket kvno 2)
0620 14:48:39.658523 (+ 257us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:39152: Request ticket server kudu/127.0.97.3@KRBTEST.COM not found in keytab (ticket kvno 2)
Metrics: {"server-negotiator.queue_time_us":435,"thread_start_us":325,"threads_started":1}
W20260620 14:48:39.658921 9705 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:39152: Request ticket server kudu/127.0.97.3@KRBTEST.COM not found in keytab (ticket kvno 2)
W20260620 14:48:39.659018 388 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskUci3DU/build/asan/bin/kudu pid 9525: Not authorized: Client connection negotiation failed: client connection to 127.0.97.3:34719: FATAL_UNAUTHORIZED: Not authorized: Request ticket server kudu/127.0.97.3@KRBTEST.COM not found in keytab (ticket kvno 2)
I20260620 14:48:39.659179 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskUci3DU/build/asan/bin/kudu with pid 9525
I20260620 14:48:39.675981 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskUci3DU/build/asan/bin/kudu pid 9174
I20260620 14:48:39.677259 9236 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 14:48:39.790468 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskUci3DU/build/asan/bin/kudu with pid 9174
2026-06-20T14:48:39Z chronyd exiting
[ OK ] SecurityITest.TestMismatchingPrincipals (21223 ms)
[ RUN ] SecurityITest.TestRequireAuthenticationInsecureCluster
2026-06-20T14:48:39Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T14:48:39Z Disabled control of system clock
I20260620 14:48:39.866604 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskUci3DU/build/asan/bin/kudu
/tmp/dist-test-taskUci3DU/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskUci3DU/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781966829091300-388-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskUci3DU/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781966829091300-388-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskUci3DU/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781966829091300-388-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskUci3DU/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781966829091300-388-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:37129
--webserver_interface=127.0.97.62
--webserver_port=0
--builtin_ntp_servers=127.0.97.20:45963
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.97.62:37129
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {}
W20260620 14:48:40.206429 9710 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 14:48:40.206959 9710 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 14:48:40.207149 9710 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 14:48:40.216197 9710 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 14:48:40.216353 9710 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 14:48:40.216441 9710 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 14:48:40.216526 9710 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 14:48:40.216609 9710 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 14:48:40.226625 9710 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:45963
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskUci3DU/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781966829091300-388-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskUci3DU/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781966829091300-388-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.97.62:37129
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:37129
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskUci3DU/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781966829091300-388-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.97.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskUci3DU/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781966829091300-388-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 13:41:20 UTC on ccycloud.quasar-vdagoi.root.comops.site
build id 4457462
ASAN enabled
I20260620 14:48:40.228197 9710 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 14:48:40.230010 9710 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 14:48:40.242578 9718 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 14:48:40.242650 9716 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 14:48:40.242617 9715 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 14:48:40.244076 9710 server_base.cc:1034] running on GCE node
I20260620 14:48:40.244958 9710 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 14:48:40.246618 9710 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 14:48:40.247866 9710 hybrid_clock.cc:648] HybridClock initialized: now 1781966920247803 us; error 65 us; skew 500 ppm
I20260620 14:48:40.248302 9710 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260620 14:48:40.251456 9710 webserver.cc:460] Webserver started at http://127.0.97.62:45389/ using document root <none> and password file <none>
I20260620 14:48:40.252301 9710 fs_manager.cc:359] Metadata directory not provided
I20260620 14:48:40.252449 9710 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 14:48:40.252784 9710 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 14:48:40.256021 9710 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskUci3DU/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781966829091300-388-0/minicluster-data/master-0/data/instance:
uuid: "adcb9e41f4ea4a7cac54e710fa91612e"
format_stamp: "Formatted at 2026-06-20 14:48:40 on dist-test-slave-q9j6"
server_key: "332308379694e19877bd15da55fd2d49"
server_key_iv: "75495e2829ca017b111ddc9b6680afac"
server_key_version: "encryptionkey@0"
I20260620 14:48:40.256839 9710 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskUci3DU/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781966829091300-388-0/minicluster-data/master-0/wal/instance:
uuid: "adcb9e41f4ea4a7cac54e710fa91612e"
format_stamp: "Formatted at 2026-06-20 14:48:40 on dist-test-slave-q9j6"
server_key: "332308379694e19877bd15da55fd2d49"
server_key_iv: "75495e2829ca017b111ddc9b6680afac"
server_key_version: "encryptionkey@0"
I20260620 14:48:40.262496 9710 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.001s
I20260620 14:48:40.266378 9724 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 14:48:40.267864 9710 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260620 14:48:40.268085 9710 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskUci3DU/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781966829091300-388-0/minicluster-data/master-0/data,/tmp/dist-test-taskUci3DU/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781966829091300-388-0/minicluster-data/master-0/wal
uuid: "adcb9e41f4ea4a7cac54e710fa91612e"
format_stamp: "Formatted at 2026-06-20 14:48:40 on dist-test-slave-q9j6"
server_key: "332308379694e19877bd15da55fd2d49"
server_key_iv: "75495e2829ca017b111ddc9b6680afac"
server_key_version: "encryptionkey@0"
I20260620 14:48:40.268330 9710 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskUci3DU/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781966829091300-388-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskUci3DU/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781966829091300-388-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskUci3DU/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781966829091300-388-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 14:48:40.284922 9710 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 14:48:40.285825 9710 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 14:48:40.286130 9710 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 14:48:40.304821 9710 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.62:37129
I20260620 14:48:40.304839 9775 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.62:37129 every 8 connection(s)
I20260620 14:48:40.306253 9710 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskUci3DU/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781966829091300-388-0/minicluster-data/master-0/data/info.pb
I20260620 14:48:40.309501 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskUci3DU/build/asan/bin/kudu as pid 9710
I20260620 14:48:40.309724 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskUci3DU/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781966829091300-388-0/minicluster-data/master-0/wal/instance
I20260620 14:48:40.310212 9776 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 14:48:40.310410 388 external_mini_cluster.cc:1428] Setting key 1909221dbcbecbb25d973ff07fd70763
I20260620 14:48:40.323750 9776 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P adcb9e41f4ea4a7cac54e710fa91612e: Bootstrap starting.
I20260620 14:48:40.324407 9779 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:48:40.312383 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51406 (local address 127.0.97.62:37129)
0620 14:48:40.313413 (+ 1030us) server_negotiation.cc:206] Beginning negotiation
0620 14:48:40.313424 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0620 14:48:40.313466 (+ 42us) server_negotiation.cc:407] Connection header received
0620 14:48:40.315085 (+ 1619us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:48:40.315111 (+ 26us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:48:40.315708 (+ 597us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:48:40.316188 (+ 480us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:48:40.316836 (+ 648us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:48:40.318160 (+ 1324us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:48:40.319520 (+ 1360us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:48:40.320326 (+ 806us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:48:40.320575 (+ 249us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:48:40.321875 (+ 1300us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 14:48:40.321883 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 14:48:40.321900 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 14:48:40.322471 (+ 571us) server_negotiation.cc:922] calling sasl_server_start()
0620 14:48:40.322552 (+ 81us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 14:48:40.322573 (+ 21us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 14:48:40.322713 (+ 140us) server_negotiation.cc:1035] Waiting for connection context
0620 14:48:40.322957 (+ 244us) server_negotiation.cc:299] Negotiation successful
0620 14:48:40.323291 (+ 334us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":571,"thread_start_us":341,"threads_started":1}
I20260620 14:48:40.327550 9776 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P adcb9e41f4ea4a7cac54e710fa91612e: Neither blocks nor log segments found. Creating new log.
I20260620 14:48:40.328611 9776 log.cc:824] T 00000000000000000000000000000000 P adcb9e41f4ea4a7cac54e710fa91612e: Log is configured to *not* fsync() on all Append() calls
I20260620 14:48:40.331398 9776 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P adcb9e41f4ea4a7cac54e710fa91612e: No bootstrap required, opened a new log
I20260620 14:48:40.339033 9776 raft_consensus.cc:348] T 00000000000000000000000000000000 P adcb9e41f4ea4a7cac54e710fa91612e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "adcb9e41f4ea4a7cac54e710fa91612e" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 37129 } }
I20260620 14:48:40.339395 9776 raft_consensus.cc:374] T 00000000000000000000000000000000 P adcb9e41f4ea4a7cac54e710fa91612e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 14:48:40.339473 9776 raft_consensus.cc:729] T 00000000000000000000000000000000 P adcb9e41f4ea4a7cac54e710fa91612e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: adcb9e41f4ea4a7cac54e710fa91612e, State: Initialized, Role: FOLLOWER
I20260620 14:48:40.340179 9776 consensus_queue.cc:260] T 00000000000000000000000000000000 P adcb9e41f4ea4a7cac54e710fa91612e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "adcb9e41f4ea4a7cac54e710fa91612e" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 37129 } }
I20260620 14:48:40.340387 9776 raft_consensus.cc:388] T 00000000000000000000000000000000 P adcb9e41f4ea4a7cac54e710fa91612e [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 14:48:40.340488 9776 raft_consensus.cc:482] T 00000000000000000000000000000000 P adcb9e41f4ea4a7cac54e710fa91612e [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 14:48:40.340651 9776 raft_consensus.cc:3037] T 00000000000000000000000000000000 P adcb9e41f4ea4a7cac54e710fa91612e [term 0 FOLLOWER]: Advancing to term 1
I20260620 14:48:40.343084 9776 raft_consensus.cc:504] T 00000000000000000000000000000000 P adcb9e41f4ea4a7cac54e710fa91612e [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "adcb9e41f4ea4a7cac54e710fa91612e" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 37129 } }
I20260620 14:48:40.343555 9776 leader_election.cc:302] T 00000000000000000000000000000000 P adcb9e41f4ea4a7cac54e710fa91612e [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: adcb9e41f4ea4a7cac54e710fa91612e; no voters:
I20260620 14:48:40.344151 9776 leader_election.cc:288] T 00000000000000000000000000000000 P adcb9e41f4ea4a7cac54e710fa91612e [CANDIDATE]: Term 1 election: Requested vote from peers
I20260620 14:48:40.344322 9781 raft_consensus.cc:2781] T 00000000000000000000000000000000 P adcb9e41f4ea4a7cac54e710fa91612e [term 1 FOLLOWER]: Leader election won for term 1
I20260620 14:48:40.344998 9781 raft_consensus.cc:686] T 00000000000000000000000000000000 P adcb9e41f4ea4a7cac54e710fa91612e [term 1 LEADER]: Becoming Leader. State: Replica: adcb9e41f4ea4a7cac54e710fa91612e, State: Running, Role: LEADER
I20260620 14:48:40.345602 9781 consensus_queue.cc:237] T 00000000000000000000000000000000 P adcb9e41f4ea4a7cac54e710fa91612e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "adcb9e41f4ea4a7cac54e710fa91612e" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 37129 } }
I20260620 14:48:40.346439 9776 sys_catalog.cc:564] T 00000000000000000000000000000000 P adcb9e41f4ea4a7cac54e710fa91612e [sys.catalog]: configured and running, proceeding with master startup.
I20260620 14:48:40.348572 9782 sys_catalog.cc:455] T 00000000000000000000000000000000 P adcb9e41f4ea4a7cac54e710fa91612e [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "adcb9e41f4ea4a7cac54e710fa91612e" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "adcb9e41f4ea4a7cac54e710fa91612e" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 37129 } } }
I20260620 14:48:40.348544 9783 sys_catalog.cc:455] T 00000000000000000000000000000000 P adcb9e41f4ea4a7cac54e710fa91612e [sys.catalog]: SysCatalogTable state changed. Reason: New leader adcb9e41f4ea4a7cac54e710fa91612e. Latest consensus state: current_term: 1 leader_uuid: "adcb9e41f4ea4a7cac54e710fa91612e" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "adcb9e41f4ea4a7cac54e710fa91612e" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 37129 } } }
I20260620 14:48:40.348958 9782 sys_catalog.cc:458] T 00000000000000000000000000000000 P adcb9e41f4ea4a7cac54e710fa91612e [sys.catalog]: This master's current role is: LEADER
I20260620 14:48:40.349041 9783 sys_catalog.cc:458] T 00000000000000000000000000000000 P adcb9e41f4ea4a7cac54e710fa91612e [sys.catalog]: This master's current role is: LEADER
I20260620 14:48:40.350142 9787 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 14:48:40.355866 9787 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 14:48:40.364964 9787 catalog_manager.cc:1347] Generated new cluster ID: 50d1ef90c6164e839b50fe12fb027e31
I20260620 14:48:40.365083 9787 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 14:48:40.389935 9787 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 14:48:40.391785 9787 catalog_manager.cc:1504] Loading token signing keys...
I20260620 14:48:40.401127 9787 catalog_manager.cc:5868] T 00000000000000000000000000000000 P adcb9e41f4ea4a7cac54e710fa91612e: Generated new TSK 0
I20260620 14:48:40.402225 9787 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260620 14:48:40.411535 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskUci3DU/build/asan/bin/kudu
/tmp/dist-test-taskUci3DU/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskUci3DU/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781966829091300-388-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskUci3DU/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781966829091300-388-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskUci3DU/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781966829091300-388-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskUci3DU/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781966829091300-388-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.1:0
--local_ip_for_outbound_sockets=127.0.97.1
--webserver_interface=127.0.97.1
--webserver_port=0
--tserver_master_addrs=127.0.97.62:37129
--builtin_ntp_servers=127.0.97.20:45963
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {}
W20260620 14:48:40.751799 9800 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 14:48:40.752135 9800 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 14:48:40.752292 9800 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 14:48:40.760139 9800 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 14:48:40.760247 9800 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 14:48:40.760429 9800 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.1
I20260620 14:48:40.769701 9800 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:45963
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskUci3DU/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781966829091300-388-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskUci3DU/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781966829091300-388-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.1:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskUci3DU/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781966829091300-388-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.97.1
--webserver_port=0
--tserver_master_addrs=127.0.97.62:37129
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.1
--log_dir=/tmp/dist-test-taskUci3DU/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781966829091300-388-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 13:41:20 UTC on ccycloud.quasar-vdagoi.root.comops.site
build id 4457462
ASAN enabled
I20260620 14:48:40.771101 9800 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 14:48:40.773270 9800 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 14:48:40.786651 9805 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 14:48:40.786898 9806 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 14:48:40.786628 9808 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 14:48:40.787393 9800 server_base.cc:1034] running on GCE node
I20260620 14:48:40.788112 9800 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 14:48:40.789076 9800 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 14:48:40.790310 9800 hybrid_clock.cc:648] HybridClock initialized: now 1781966920790250 us; error 53 us; skew 500 ppm
I20260620 14:48:40.790649 9800 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260620 14:48:40.797123 9800 webserver.cc:460] Webserver started at http://127.0.97.1:35783/ using document root <none> and password file <none>
I20260620 14:48:40.797915 9800 fs_manager.cc:359] Metadata directory not provided
I20260620 14:48:40.798051 9800 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 14:48:40.798399 9800 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 14:48:40.801397 9800 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskUci3DU/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781966829091300-388-0/minicluster-data/ts-0/data/instance:
uuid: "46cf21b94ec24fc1b17984d4911a286e"
format_stamp: "Formatted at 2026-06-20 14:48:40 on dist-test-slave-q9j6"
server_key: "1d742be879b2fc2d1c929a29dd09f98e"
server_key_iv: "29cb77a88f3768d73b4459cfb793c7c5"
server_key_version: "encryptionkey@0"
I20260620 14:48:40.802240 9800 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskUci3DU/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781966829091300-388-0/minicluster-data/ts-0/wal/instance:
uuid: "46cf21b94ec24fc1b17984d4911a286e"
format_stamp: "Formatted at 2026-06-20 14:48:40 on dist-test-slave-q9j6"
server_key: "1d742be879b2fc2d1c929a29dd09f98e"
server_key_iv: "29cb77a88f3768d73b4459cfb793c7c5"
server_key_version: "encryptionkey@0"
I20260620 14:48:40.807551 9800 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.007s sys 0.000s
I20260620 14:48:40.811117 9814 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 14:48:40.812568 9800 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260620 14:48:40.812763 9800 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskUci3DU/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781966829091300-388-0/minicluster-data/ts-0/data,/tmp/dist-test-taskUci3DU/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781966829091300-388-0/minicluster-data/ts-0/wal
uuid: "46cf21b94ec24fc1b17984d4911a286e"
format_stamp: "Formatted at 2026-06-20 14:48:40 on dist-test-slave-q9j6"
server_key: "1d742be879b2fc2d1c929a29dd09f98e"
server_key_iv: "29cb77a88f3768d73b4459cfb793c7c5"
server_key_version: "encryptionkey@0"
I20260620 14:48:40.812988 9800 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskUci3DU/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781966829091300-388-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskUci3DU/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781966829091300-388-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskUci3DU/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781966829091300-388-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 14:48:40.837795 9800 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 14:48:40.838706 9800 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 14:48:40.838995 9800 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 14:48:40.840046 9800 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 14:48:40.841554 9800 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 14:48:40.841686 9800 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 14:48:40.841806 9800 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 14:48:40.841890 9800 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 14:48:40.875062 9800 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.1:33249
I20260620 14:48:40.875152 9926 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.1:33249 every 8 connection(s)
I20260620 14:48:40.876379 9800 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskUci3DU/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781966829091300-388-0/minicluster-data/ts-0/data/info.pb
I20260620 14:48:40.877028 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskUci3DU/build/asan/bin/kudu as pid 9800
I20260620 14:48:40.877254 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskUci3DU/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781966829091300-388-0/minicluster-data/ts-0/wal/instance
I20260620 14:48:40.878075 388 external_mini_cluster.cc:1428] Setting key 375e01c25398d60736b8b003f723d3a4
I20260620 14:48:40.882035 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskUci3DU/build/asan/bin/kudu
/tmp/dist-test-taskUci3DU/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskUci3DU/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781966829091300-388-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskUci3DU/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781966829091300-388-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskUci3DU/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781966829091300-388-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskUci3DU/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781966829091300-388-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.2:0
--local_ip_for_outbound_sockets=127.0.97.2
--webserver_interface=127.0.97.2
--webserver_port=0
--tserver_master_addrs=127.0.97.62:37129
--builtin_ntp_servers=127.0.97.20:45963
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {}
I20260620 14:48:40.890183 9929 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:48:40.879159 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.1:56527 (local address 127.0.97.62:37129)
0620 14:48:40.879648 (+ 489us) server_negotiation.cc:206] Beginning negotiation
0620 14:48:40.879654 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 14:48:40.881318 (+ 1664us) server_negotiation.cc:407] Connection header received
0620 14:48:40.883138 (+ 1820us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:48:40.883147 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:48:40.883242 (+ 95us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:48:40.883364 (+ 122us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:48:40.884454 (+ 1090us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:48:40.885613 (+ 1159us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:48:40.887113 (+ 1500us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:48:40.887916 (+ 803us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:48:40.888063 (+ 147us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:48:40.889448 (+ 1385us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 14:48:40.889454 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 14:48:40.889457 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 14:48:40.889476 (+ 19us) server_negotiation.cc:922] calling sasl_server_start()
0620 14:48:40.889522 (+ 46us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 14:48:40.889534 (+ 12us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 14:48:40.889604 (+ 70us) server_negotiation.cc:1035] Waiting for connection context
0620 14:48:40.889852 (+ 248us) server_negotiation.cc:299] Negotiation successful
0620 14:48:40.889928 (+ 76us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":352,"thread_start_us":240,"threads_started":1}
I20260620 14:48:40.891760 9930 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:48:40.879911 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:37129 (local address 127.0.97.1:56527)
0620 14:48:40.881145 (+ 1234us) negotiation.cc:107] Waiting for socket to connect
0620 14:48:40.881190 (+ 45us) client_negotiation.cc:174] Beginning negotiation
0620 14:48:40.882851 (+ 1661us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 14:48:40.883560 (+ 709us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 14:48:40.883568 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 14:48:40.883722 (+ 154us) client_negotiation.cc:189] Negotiated authn=SASL
0620 14:48:40.884281 (+ 559us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:48:40.884297 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:48:40.885758 (+ 1461us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:48:40.885762 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:48:40.886967 (+ 1205us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:48:40.886978 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:48:40.888117 (+ 1139us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:48:40.888122 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:48:40.888315 (+ 193us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:48:40.889065 (+ 750us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0620 14:48:40.889071 (+ 6us) client_negotiation.cc:656] calling sasl_client_start()
0620 14:48:40.889134 (+ 63us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0620 14:48:40.889246 (+ 112us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 14:48:40.889632 (+ 386us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 14:48:40.889645 (+ 13us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 14:48:40.889652 (+ 7us) client_negotiation.cc:769] Sending connection context
0620 14:48:40.889839 (+ 187us) client_negotiation.cc:240] Negotiation successful
0620 14:48:40.890264 (+ 425us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":697,"thread_start_us":436,"threads_started":1}
I20260620 14:48:40.893297 9927 heartbeater.cc:347] Connected to a master server at 127.0.97.62:37129
I20260620 14:48:40.893620 9927 heartbeater.cc:464] Registering TS with master...
I20260620 14:48:40.894371 9927 heartbeater.cc:511] Master 127.0.97.62:37129 requested a full tablet report, sending...
I20260620 14:48:40.896498 9741 ts_manager.cc:194] Registered new tserver with Master: 46cf21b94ec24fc1b17984d4911a286e (127.0.97.1:33249)
I20260620 14:48:40.898029 9741 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.97.1:56527
W20260620 14:48:41.220808 9931 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 14:48:41.221159 9931 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 14:48:41.221316 9931 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 14:48:41.228768 9931 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 14:48:41.228928 9931 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 14:48:41.229105 9931 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.2
I20260620 14:48:41.238718 9931 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:45963
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskUci3DU/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781966829091300-388-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskUci3DU/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781966829091300-388-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.2:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskUci3DU/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781966829091300-388-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.97.2
--webserver_port=0
--tserver_master_addrs=127.0.97.62:37129
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.2
--log_dir=/tmp/dist-test-taskUci3DU/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781966829091300-388-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 13:41:20 UTC on ccycloud.quasar-vdagoi.root.comops.site
build id 4457462
ASAN enabled
I20260620 14:48:41.240185 9931 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 14:48:41.241976 9931 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 14:48:41.255471 9936 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 14:48:41.255517 9937 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 14:48:41.255430 9939 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 14:48:41.261183 9931 server_base.cc:1034] running on GCE node
I20260620 14:48:41.261860 9931 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 14:48:41.262873 9931 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 14:48:41.264113 9931 hybrid_clock.cc:648] HybridClock initialized: now 1781966921264049 us; error 71 us; skew 500 ppm
I20260620 14:48:41.264467 9931 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260620 14:48:41.266698 9931 webserver.cc:460] Webserver started at http://127.0.97.2:34787/ using document root <none> and password file <none>
I20260620 14:48:41.267418 9931 fs_manager.cc:359] Metadata directory not provided
I20260620 14:48:41.267550 9931 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 14:48:41.267908 9931 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 14:48:41.270720 9931 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskUci3DU/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781966829091300-388-0/minicluster-data/ts-1/data/instance:
uuid: "1669664ecedd491dbb03db040b4ed62f"
format_stamp: "Formatted at 2026-06-20 14:48:41 on dist-test-slave-q9j6"
server_key: "c10f70fe983c2355ee3aea48a88d0caf"
server_key_iv: "ce4ae5a13e1b52e8d76f5fa5bdb65fac"
server_key_version: "encryptionkey@0"
I20260620 14:48:41.271512 9931 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskUci3DU/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781966829091300-388-0/minicluster-data/ts-1/wal/instance:
uuid: "1669664ecedd491dbb03db040b4ed62f"
format_stamp: "Formatted at 2026-06-20 14:48:41 on dist-test-slave-q9j6"
server_key: "c10f70fe983c2355ee3aea48a88d0caf"
server_key_iv: "ce4ae5a13e1b52e8d76f5fa5bdb65fac"
server_key_version: "encryptionkey@0"
I20260620 14:48:41.276810 9931 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.003s sys 0.001s
I20260620 14:48:41.280081 9945 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 14:48:41.281361 9931 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.002s
I20260620 14:48:41.281543 9931 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskUci3DU/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781966829091300-388-0/minicluster-data/ts-1/data,/tmp/dist-test-taskUci3DU/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781966829091300-388-0/minicluster-data/ts-1/wal
uuid: "1669664ecedd491dbb03db040b4ed62f"
format_stamp: "Formatted at 2026-06-20 14:48:41 on dist-test-slave-q9j6"
server_key: "c10f70fe983c2355ee3aea48a88d0caf"
server_key_iv: "ce4ae5a13e1b52e8d76f5fa5bdb65fac"
server_key_version: "encryptionkey@0"
I20260620 14:48:41.281759 9931 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskUci3DU/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781966829091300-388-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskUci3DU/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781966829091300-388-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskUci3DU/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781966829091300-388-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 14:48:41.297645 9931 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 14:48:41.298571 9931 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 14:48:41.298872 9931 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 14:48:41.299916 9931 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 14:48:41.301437 9931 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 14:48:41.301564 9931 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 14:48:41.301707 9931 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 14:48:41.301796 9931 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 14:48:41.335563 9931 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.2:38917
I20260620 14:48:41.335687 10057 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.2:38917 every 8 connection(s)
I20260620 14:48:41.337029 9931 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskUci3DU/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781966829091300-388-0/minicluster-data/ts-1/data/info.pb
I20260620 14:48:41.337445 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskUci3DU/build/asan/bin/kudu as pid 9931
I20260620 14:48:41.337641 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskUci3DU/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781966829091300-388-0/minicluster-data/ts-1/wal/instance
I20260620 14:48:41.338233 388 external_mini_cluster.cc:1428] Setting key eb255ad4b216097fc410c06282a72685
I20260620 14:48:41.341415 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskUci3DU/build/asan/bin/kudu
/tmp/dist-test-taskUci3DU/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskUci3DU/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781966829091300-388-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskUci3DU/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781966829091300-388-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskUci3DU/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781966829091300-388-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskUci3DU/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781966829091300-388-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.3:0
--local_ip_for_outbound_sockets=127.0.97.3
--webserver_interface=127.0.97.3
--webserver_port=0
--tserver_master_addrs=127.0.97.62:37129
--builtin_ntp_servers=127.0.97.20:45963
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {}
I20260620 14:48:41.352983 9929 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:48:41.339821 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.2:44213 (local address 127.0.97.62:37129)
0620 14:48:41.340011 (+ 190us) server_negotiation.cc:206] Beginning negotiation
0620 14:48:41.340018 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0620 14:48:41.341895 (+ 1877us) server_negotiation.cc:407] Connection header received
0620 14:48:41.344875 (+ 2980us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:48:41.344879 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:48:41.344951 (+ 72us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:48:41.345045 (+ 94us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:48:41.346718 (+ 1673us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:48:41.347660 (+ 942us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:48:41.350043 (+ 2383us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:48:41.350444 (+ 401us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:48:41.350582 (+ 138us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:48:41.352140 (+ 1558us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 14:48:41.352147 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 14:48:41.352152 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 14:48:41.352178 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0620 14:48:41.352208 (+ 30us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 14:48:41.352220 (+ 12us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 14:48:41.352319 (+ 99us) server_negotiation.cc:1035] Waiting for connection context
0620 14:48:41.352552 (+ 233us) server_negotiation.cc:299] Negotiation successful
0620 14:48:41.352662 (+ 110us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":65}
I20260620 14:48:41.354076 10060 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:48:41.340516 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:37129 (local address 127.0.97.2:44213)
0620 14:48:41.341705 (+ 1189us) negotiation.cc:107] Waiting for socket to connect
0620 14:48:41.341759 (+ 54us) client_negotiation.cc:174] Beginning negotiation
0620 14:48:41.344539 (+ 2780us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 14:48:41.345370 (+ 831us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 14:48:41.345385 (+ 15us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 14:48:41.345530 (+ 145us) client_negotiation.cc:189] Negotiated authn=SASL
0620 14:48:41.346490 (+ 960us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:48:41.346521 (+ 31us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:48:41.347838 (+ 1317us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:48:41.347846 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:48:41.349859 (+ 2013us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:48:41.349872 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:48:41.350689 (+ 817us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:48:41.350697 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:48:41.350964 (+ 267us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:48:41.351747 (+ 783us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0620 14:48:41.351751 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0620 14:48:41.351836 (+ 85us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0620 14:48:41.351934 (+ 98us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 14:48:41.352318 (+ 384us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 14:48:41.352332 (+ 14us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 14:48:41.352342 (+ 10us) client_negotiation.cc:769] Sending connection context
0620 14:48:41.352547 (+ 205us) client_negotiation.cc:240] Negotiation successful
0620 14:48:41.352968 (+ 421us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":536,"thread_start_us":316,"threads_started":1}
I20260620 14:48:41.355706 10058 heartbeater.cc:347] Connected to a master server at 127.0.97.62:37129
I20260620 14:48:41.356021 10058 heartbeater.cc:464] Registering TS with master...
I20260620 14:48:41.356840 10058 heartbeater.cc:511] Master 127.0.97.62:37129 requested a full tablet report, sending...
I20260620 14:48:41.358326 9741 ts_manager.cc:194] Registered new tserver with Master: 1669664ecedd491dbb03db040b4ed62f (127.0.97.2:38917)
I20260620 14:48:41.359229 9741 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.97.2:44213
W20260620 14:48:41.673219 10061 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 14:48:41.673578 10061 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 14:48:41.673740 10061 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 14:48:41.681200 10061 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 14:48:41.681358 10061 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 14:48:41.681542 10061 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.3
I20260620 14:48:41.691198 10061 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:45963
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskUci3DU/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781966829091300-388-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskUci3DU/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781966829091300-388-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.3:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskUci3DU/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781966829091300-388-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.97.3
--webserver_port=0
--tserver_master_addrs=127.0.97.62:37129
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.3
--log_dir=/tmp/dist-test-taskUci3DU/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781966829091300-388-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 13:41:20 UTC on ccycloud.quasar-vdagoi.root.comops.site
build id 4457462
ASAN enabled
I20260620 14:48:41.692631 10061 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 14:48:41.694638 10061 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 14:48:41.707989 10067 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 14:48:41.708060 10069 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 14:48:41.708082 10066 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 14:48:41.708729 10061 server_base.cc:1034] running on GCE node
I20260620 14:48:41.709372 10061 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 14:48:41.710362 10061 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 14:48:41.711647 10061 hybrid_clock.cc:648] HybridClock initialized: now 1781966921711538 us; error 57 us; skew 500 ppm
I20260620 14:48:41.711989 10061 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260620 14:48:41.714308 10061 webserver.cc:460] Webserver started at http://127.0.97.3:45185/ using document root <none> and password file <none>
I20260620 14:48:41.715051 10061 fs_manager.cc:359] Metadata directory not provided
I20260620 14:48:41.715188 10061 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 14:48:41.715515 10061 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 14:48:41.718582 10061 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskUci3DU/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781966829091300-388-0/minicluster-data/ts-2/data/instance:
uuid: "5edfa8b91e86440eb93d8b14b1b1d254"
format_stamp: "Formatted at 2026-06-20 14:48:41 on dist-test-slave-q9j6"
server_key: "f40aba4af588d6292e4aadcba1ca50f9"
server_key_iv: "ac6a5edcd33d38fdcd586dccc6965887"
server_key_version: "encryptionkey@0"
I20260620 14:48:41.719393 10061 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskUci3DU/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781966829091300-388-0/minicluster-data/ts-2/wal/instance:
uuid: "5edfa8b91e86440eb93d8b14b1b1d254"
format_stamp: "Formatted at 2026-06-20 14:48:41 on dist-test-slave-q9j6"
server_key: "f40aba4af588d6292e4aadcba1ca50f9"
server_key_iv: "ac6a5edcd33d38fdcd586dccc6965887"
server_key_version: "encryptionkey@0"
I20260620 14:48:41.724666 10061 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.003s sys 0.001s
I20260620 14:48:41.728221 10075 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 14:48:41.729627 10061 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260620 14:48:41.729822 10061 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskUci3DU/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781966829091300-388-0/minicluster-data/ts-2/data,/tmp/dist-test-taskUci3DU/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781966829091300-388-0/minicluster-data/ts-2/wal
uuid: "5edfa8b91e86440eb93d8b14b1b1d254"
format_stamp: "Formatted at 2026-06-20 14:48:41 on dist-test-slave-q9j6"
server_key: "f40aba4af588d6292e4aadcba1ca50f9"
server_key_iv: "ac6a5edcd33d38fdcd586dccc6965887"
server_key_version: "encryptionkey@0"
I20260620 14:48:41.730065 10061 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskUci3DU/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781966829091300-388-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskUci3DU/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781966829091300-388-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskUci3DU/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781966829091300-388-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 14:48:41.744650 10061 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 14:48:41.745549 10061 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 14:48:41.745843 10061 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 14:48:41.746853 10061 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 14:48:41.748617 10061 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 14:48:41.748737 10061 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 14:48:41.748875 10061 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 14:48:41.748963 10061 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 14:48:41.782640 10061 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.3:35577
I20260620 14:48:41.782694 10187 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.3:35577 every 8 connection(s)
I20260620 14:48:41.784152 10061 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskUci3DU/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781966829091300-388-0/minicluster-data/ts-2/data/info.pb
I20260620 14:48:41.787472 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskUci3DU/build/asan/bin/kudu as pid 10061
I20260620 14:48:41.787679 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskUci3DU/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781966829091300-388-0/minicluster-data/ts-2/wal/instance
I20260620 14:48:41.788293 388 external_mini_cluster.cc:1428] Setting key de209060dfa2fc03046087e18be07ad3
I20260620 14:48:41.797638 9929 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:48:41.786851 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.3:51327 (local address 127.0.97.62:37129)
0620 14:48:41.787281 (+ 430us) server_negotiation.cc:206] Beginning negotiation
0620 14:48:41.787290 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0620 14:48:41.789045 (+ 1755us) server_negotiation.cc:407] Connection header received
0620 14:48:41.791348 (+ 2303us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:48:41.791354 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:48:41.791441 (+ 87us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:48:41.791567 (+ 126us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:48:41.792751 (+ 1184us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:48:41.793456 (+ 705us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:48:41.794992 (+ 1536us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:48:41.795380 (+ 388us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:48:41.795473 (+ 93us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:48:41.796939 (+ 1466us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 14:48:41.796944 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 14:48:41.796947 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 14:48:41.796964 (+ 17us) server_negotiation.cc:922] calling sasl_server_start()
0620 14:48:41.796994 (+ 30us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 14:48:41.797007 (+ 13us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 14:48:41.797080 (+ 73us) server_negotiation.cc:1035] Waiting for connection context
0620 14:48:41.797310 (+ 230us) server_negotiation.cc:299] Negotiation successful
0620 14:48:41.797388 (+ 78us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":273}
I20260620 14:48:41.798679 10190 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:48:41.787679 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:37129 (local address 127.0.97.3:51327)
0620 14:48:41.788858 (+ 1179us) negotiation.cc:107] Waiting for socket to connect
0620 14:48:41.788898 (+ 40us) client_negotiation.cc:174] Beginning negotiation
0620 14:48:41.791036 (+ 2138us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 14:48:41.791886 (+ 850us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 14:48:41.791894 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 14:48:41.792005 (+ 111us) client_negotiation.cc:189] Negotiated authn=SASL
0620 14:48:41.792604 (+ 599us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:48:41.792621 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:48:41.793593 (+ 972us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:48:41.793596 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:48:41.794854 (+ 1258us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:48:41.794866 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:48:41.795569 (+ 703us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:48:41.795574 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:48:41.795989 (+ 415us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:48:41.796632 (+ 643us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0620 14:48:41.796634 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0620 14:48:41.796676 (+ 42us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0620 14:48:41.796757 (+ 81us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 14:48:41.797108 (+ 351us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 14:48:41.797115 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 14:48:41.797121 (+ 6us) client_negotiation.cc:769] Sending connection context
0620 14:48:41.797265 (+ 144us) client_negotiation.cc:240] Negotiation successful
0620 14:48:41.797572 (+ 307us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":559,"thread_start_us":302,"threads_started":1}
I20260620 14:48:41.800109 10188 heartbeater.cc:347] Connected to a master server at 127.0.97.62:37129
I20260620 14:48:41.800422 10188 heartbeater.cc:464] Registering TS with master...
I20260620 14:48:41.801192 10188 heartbeater.cc:511] Master 127.0.97.62:37129 requested a full tablet report, sending...
I20260620 14:48:41.802815 9741 ts_manager.cc:194] Registered new tserver with Master: 5edfa8b91e86440eb93d8b14b1b1d254 (127.0.97.3:35577)
I20260620 14:48:41.803802 9741 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.97.3:51327
I20260620 14:48:41.804276 388 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
W20260620 14:48:41.810391 9929 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 14:48:41.808719 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51416 (local address 127.0.97.62:37129)
0620 14:48:41.808914 (+ 195us) server_negotiation.cc:206] Beginning negotiation
0620 14:48:41.808919 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 14:48:41.809461 (+ 542us) server_negotiation.cc:407] Connection header received
0620 14:48:41.809607 (+ 146us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:48:41.809611 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:48:41.809686 (+ 75us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:48:41.809776 (+ 90us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:48:41.810121 (+ 345us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:51416: BlockingRecv error: recv got EOF from 127.0.0.1:51416 (error 108)
Metrics: {"server-negotiator.queue_time_us":65}
I20260620 14:48:41.811160 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskUci3DU/build/asan/bin/kudu pid 9800
I20260620 14:48:41.819362 10197 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:48:41.812164 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58792 (local address 127.0.97.1:33249)
0620 14:48:41.812723 (+ 559us) server_negotiation.cc:206] Beginning negotiation
0620 14:48:41.812731 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0620 14:48:41.812756 (+ 25us) server_negotiation.cc:407] Connection header received
0620 14:48:41.812810 (+ 54us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:48:41.812816 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:48:41.813056 (+ 240us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:48:41.813227 (+ 171us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:48:41.813624 (+ 397us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:48:41.814623 (+ 999us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:48:41.815930 (+ 1307us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:48:41.816590 (+ 660us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:48:41.816722 (+ 132us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:48:41.817812 (+ 1090us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 14:48:41.817823 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 14:48:41.817846 (+ 23us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 14:48:41.818459 (+ 613us) server_negotiation.cc:922] calling sasl_server_start()
0620 14:48:41.818506 (+ 47us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 14:48:41.818526 (+ 20us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 14:48:41.818691 (+ 165us) server_negotiation.cc:1035] Waiting for connection context
0620 14:48:41.818866 (+ 175us) server_negotiation.cc:299] Negotiation successful
0620 14:48:41.819003 (+ 137us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":401,"thread_start_us":261,"threads_started":1}
I20260620 14:48:41.820727 9922 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 14:48:41.901890 9927 heartbeater.cc:503] Master 127.0.97.62:37129 was elected leader, sending a full tablet report...
I20260620 14:48:41.904484 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskUci3DU/build/asan/bin/kudu with pid 9800
I20260620 14:48:41.920953 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskUci3DU/build/asan/bin/kudu pid 9931
I20260620 14:48:41.928900 10199 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:48:41.921797 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45130 (local address 127.0.97.2:38917)
0620 14:48:41.922324 (+ 527us) server_negotiation.cc:206] Beginning negotiation
0620 14:48:41.922338 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0620 14:48:41.922366 (+ 28us) server_negotiation.cc:407] Connection header received
0620 14:48:41.922432 (+ 66us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:48:41.922439 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:48:41.922663 (+ 224us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:48:41.922850 (+ 187us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:48:41.923280 (+ 430us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:48:41.924358 (+ 1078us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:48:41.925612 (+ 1254us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:48:41.926212 (+ 600us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:48:41.926350 (+ 138us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:48:41.927380 (+ 1030us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 14:48:41.927388 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 14:48:41.927402 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 14:48:41.928081 (+ 679us) server_negotiation.cc:922] calling sasl_server_start()
0620 14:48:41.928132 (+ 51us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 14:48:41.928151 (+ 19us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 14:48:41.928276 (+ 125us) server_negotiation.cc:1035] Waiting for connection context
0620 14:48:41.928425 (+ 149us) server_negotiation.cc:299] Negotiation successful
0620 14:48:41.928574 (+ 149us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":398,"thread_start_us":271,"threads_started":1}
I20260620 14:48:41.930192 10053 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 14:48:42.013104 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskUci3DU/build/asan/bin/kudu with pid 9931
I20260620 14:48:42.028939 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskUci3DU/build/asan/bin/kudu pid 10061
I20260620 14:48:42.037410 10201 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:48:42.029995 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56634 (local address 127.0.97.3:35577)
0620 14:48:42.030559 (+ 564us) server_negotiation.cc:206] Beginning negotiation
0620 14:48:42.030566 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0620 14:48:42.030598 (+ 32us) server_negotiation.cc:407] Connection header received
0620 14:48:42.030657 (+ 59us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:48:42.030664 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:48:42.030882 (+ 218us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:48:42.031053 (+ 171us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:48:42.031488 (+ 435us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:48:42.032586 (+ 1098us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:48:42.033883 (+ 1297us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:48:42.034461 (+ 578us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:48:42.034649 (+ 188us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:48:42.035934 (+ 1285us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 14:48:42.035943 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 14:48:42.035958 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 14:48:42.036579 (+ 621us) server_negotiation.cc:922] calling sasl_server_start()
0620 14:48:42.036629 (+ 50us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 14:48:42.036648 (+ 19us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 14:48:42.036790 (+ 142us) server_negotiation.cc:1035] Waiting for connection context
0620 14:48:42.036939 (+ 149us) server_negotiation.cc:299] Negotiation successful
0620 14:48:42.037072 (+ 133us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":416,"thread_start_us":263,"threads_started":1}
I20260620 14:48:42.038861 10183 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 14:48:42.122879 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskUci3DU/build/asan/bin/kudu with pid 10061
I20260620 14:48:42.138520 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskUci3DU/build/asan/bin/kudu pid 9710
I20260620 14:48:42.139820 9771 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 14:48:42.243196 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskUci3DU/build/asan/bin/kudu with pid 9710
2026-06-20T14:48:42Z chronyd exiting
[ OK ] SecurityITest.TestRequireAuthenticationInsecureCluster (2447 ms)
[ RUN ] SecurityITest.TestRequireEncryptionInsecureCluster
2026-06-20T14:48:42Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T14:48:42Z Disabled control of system clock
I20260620 14:48:42.315310 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskUci3DU/build/asan/bin/kudu
/tmp/dist-test-taskUci3DU/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskUci3DU/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781966829091300-388-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskUci3DU/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781966829091300-388-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskUci3DU/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781966829091300-388-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskUci3DU/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781966829091300-388-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:37905
--webserver_interface=127.0.97.62
--webserver_port=0
--builtin_ntp_servers=127.0.97.20:41805
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.97.62:37905
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc_encryption=disabled
--rpc_authentication=disabled with env {}
W20260620 14:48:42.654606 10207 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 14:48:42.654932 10207 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 14:48:42.655019 10207 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 14:48:42.662559 10207 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 14:48:42.662683 10207 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 14:48:42.662720 10207 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 14:48:42.662770 10207 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 14:48:42.662814 10207 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 14:48:42.672266 10207 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:41805
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskUci3DU/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781966829091300-388-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskUci3DU/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781966829091300-388-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.97.62:37905
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:37905
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskUci3DU/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781966829091300-388-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.97.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskUci3DU/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781966829091300-388-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 13:41:20 UTC on ccycloud.quasar-vdagoi.root.comops.site
build id 4457462
ASAN enabled
I20260620 14:48:42.673583 10207 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 14:48:42.675426 10207 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 14:48:42.687633 10212 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 14:48:42.687667 10215 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 14:48:42.687702 10213 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 14:48:42.688627 10207 server_base.cc:1034] running on GCE node
I20260620 14:48:42.689388 10207 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 14:48:42.690835 10207 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 14:48:42.692077 10207 hybrid_clock.cc:648] HybridClock initialized: now 1781966922692048 us; error 67 us; skew 500 ppm
I20260620 14:48:42.692466 10207 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260620 14:48:42.694826 10207 webserver.cc:460] Webserver started at http://127.0.97.62:39969/ using document root <none> and password file <none>
I20260620 14:48:42.695585 10207 fs_manager.cc:359] Metadata directory not provided
I20260620 14:48:42.695792 10207 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 14:48:42.696141 10207 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 14:48:42.699168 10207 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskUci3DU/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781966829091300-388-0/minicluster-data/master-0/data/instance:
uuid: "6c42ba7d4c714a91996f66f172fa8c4d"
format_stamp: "Formatted at 2026-06-20 14:48:42 on dist-test-slave-q9j6"
server_key: "2e676672bb9ac0b90f770e283cc864c2"
server_key_iv: "f1762eae95ffa47c4d63f13539265fa4"
server_key_version: "encryptionkey@0"
I20260620 14:48:42.700047 10207 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskUci3DU/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781966829091300-388-0/minicluster-data/master-0/wal/instance:
uuid: "6c42ba7d4c714a91996f66f172fa8c4d"
format_stamp: "Formatted at 2026-06-20 14:48:42 on dist-test-slave-q9j6"
server_key: "2e676672bb9ac0b90f770e283cc864c2"
server_key_iv: "f1762eae95ffa47c4d63f13539265fa4"
server_key_version: "encryptionkey@0"
I20260620 14:48:42.705588 10207 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.003s sys 0.002s
I20260620 14:48:42.709396 10221 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 14:48:42.710819 10207 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260620 14:48:42.711022 10207 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskUci3DU/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781966829091300-388-0/minicluster-data/master-0/data,/tmp/dist-test-taskUci3DU/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781966829091300-388-0/minicluster-data/master-0/wal
uuid: "6c42ba7d4c714a91996f66f172fa8c4d"
format_stamp: "Formatted at 2026-06-20 14:48:42 on dist-test-slave-q9j6"
server_key: "2e676672bb9ac0b90f770e283cc864c2"
server_key_iv: "f1762eae95ffa47c4d63f13539265fa4"
server_key_version: "encryptionkey@0"
I20260620 14:48:42.711254 10207 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskUci3DU/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781966829091300-388-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskUci3DU/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781966829091300-388-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskUci3DU/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781966829091300-388-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 14:48:42.717195 10207 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 14:48:42.718037 10207 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 14:48:42.718283 10207 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 14:48:42.736765 10207 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.62:37905
I20260620 14:48:42.736771 10272 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.62:37905 every 8 connection(s)
I20260620 14:48:42.738176 10207 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskUci3DU/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781966829091300-388-0/minicluster-data/master-0/data/info.pb
I20260620 14:48:42.741955 10273 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 14:48:42.748214 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskUci3DU/build/asan/bin/kudu as pid 10207
I20260620 14:48:42.748413 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskUci3DU/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781966829091300-388-0/minicluster-data/master-0/wal/instance
I20260620 14:48:42.749075 388 external_mini_cluster.cc:1428] Setting key 044d4c5891b0ea93255d240216e24ee8
I20260620 14:48:42.752640 10273 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 6c42ba7d4c714a91996f66f172fa8c4d: Bootstrap starting.
I20260620 14:48:42.756273 10273 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 6c42ba7d4c714a91996f66f172fa8c4d: Neither blocks nor log segments found. Creating new log.
I20260620 14:48:42.757464 10273 log.cc:824] T 00000000000000000000000000000000 P 6c42ba7d4c714a91996f66f172fa8c4d: Log is configured to *not* fsync() on all Append() calls
I20260620 14:48:42.757908 10276 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:48:42.751321 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37610 (local address 127.0.97.62:37905)
0620 14:48:42.752351 (+ 1030us) server_negotiation.cc:206] Beginning negotiation
0620 14:48:42.752362 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0620 14:48:42.752402 (+ 40us) server_negotiation.cc:407] Connection header received
0620 14:48:42.754137 (+ 1735us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:48:42.754164 (+ 27us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:48:42.754609 (+ 445us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:48:42.754946 (+ 337us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:48:42.756164 (+ 1218us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 14:48:42.756170 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 14:48:42.756187 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 14:48:42.756200 (+ 13us) server_negotiation.cc:922] calling sasl_server_start()
0620 14:48:42.756262 (+ 62us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 14:48:42.756277 (+ 15us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 14:48:42.756459 (+ 182us) server_negotiation.cc:1035] Waiting for connection context
0620 14:48:42.756602 (+ 143us) server_negotiation.cc:299] Negotiation successful
0620 14:48:42.756914 (+ 312us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":576,"thread_start_us":376,"threads_started":1}
I20260620 14:48:42.760830 10273 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 6c42ba7d4c714a91996f66f172fa8c4d: No bootstrap required, opened a new log
I20260620 14:48:42.768054 10273 raft_consensus.cc:348] T 00000000000000000000000000000000 P 6c42ba7d4c714a91996f66f172fa8c4d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6c42ba7d4c714a91996f66f172fa8c4d" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 37905 } }
I20260620 14:48:42.768433 10273 raft_consensus.cc:374] T 00000000000000000000000000000000 P 6c42ba7d4c714a91996f66f172fa8c4d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 14:48:42.768539 10273 raft_consensus.cc:729] T 00000000000000000000000000000000 P 6c42ba7d4c714a91996f66f172fa8c4d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6c42ba7d4c714a91996f66f172fa8c4d, State: Initialized, Role: FOLLOWER
I20260620 14:48:42.769184 10273 consensus_queue.cc:260] T 00000000000000000000000000000000 P 6c42ba7d4c714a91996f66f172fa8c4d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6c42ba7d4c714a91996f66f172fa8c4d" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 37905 } }
I20260620 14:48:42.769464 10273 raft_consensus.cc:388] T 00000000000000000000000000000000 P 6c42ba7d4c714a91996f66f172fa8c4d [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 14:48:42.769619 10273 raft_consensus.cc:482] T 00000000000000000000000000000000 P 6c42ba7d4c714a91996f66f172fa8c4d [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 14:48:42.769795 10273 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 6c42ba7d4c714a91996f66f172fa8c4d [term 0 FOLLOWER]: Advancing to term 1
I20260620 14:48:42.771973 10273 raft_consensus.cc:504] T 00000000000000000000000000000000 P 6c42ba7d4c714a91996f66f172fa8c4d [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6c42ba7d4c714a91996f66f172fa8c4d" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 37905 } }
I20260620 14:48:42.772457 10273 leader_election.cc:302] T 00000000000000000000000000000000 P 6c42ba7d4c714a91996f66f172fa8c4d [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 6c42ba7d4c714a91996f66f172fa8c4d; no voters:
I20260620 14:48:42.773047 10273 leader_election.cc:288] T 00000000000000000000000000000000 P 6c42ba7d4c714a91996f66f172fa8c4d [CANDIDATE]: Term 1 election: Requested vote from peers
I20260620 14:48:42.773181 10278 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 6c42ba7d4c714a91996f66f172fa8c4d [term 1 FOLLOWER]: Leader election won for term 1
I20260620 14:48:42.774245 10278 raft_consensus.cc:686] T 00000000000000000000000000000000 P 6c42ba7d4c714a91996f66f172fa8c4d [term 1 LEADER]: Becoming Leader. State: Replica: 6c42ba7d4c714a91996f66f172fa8c4d, State: Running, Role: LEADER
I20260620 14:48:42.774739 10278 consensus_queue.cc:237] T 00000000000000000000000000000000 P 6c42ba7d4c714a91996f66f172fa8c4d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6c42ba7d4c714a91996f66f172fa8c4d" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 37905 } }
I20260620 14:48:42.775056 10273 sys_catalog.cc:564] T 00000000000000000000000000000000 P 6c42ba7d4c714a91996f66f172fa8c4d [sys.catalog]: configured and running, proceeding with master startup.
I20260620 14:48:42.777486 10280 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6c42ba7d4c714a91996f66f172fa8c4d [sys.catalog]: SysCatalogTable state changed. Reason: New leader 6c42ba7d4c714a91996f66f172fa8c4d. Latest consensus state: current_term: 1 leader_uuid: "6c42ba7d4c714a91996f66f172fa8c4d" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6c42ba7d4c714a91996f66f172fa8c4d" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 37905 } } }
I20260620 14:48:42.777906 10280 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6c42ba7d4c714a91996f66f172fa8c4d [sys.catalog]: This master's current role is: LEADER
I20260620 14:48:42.777832 10279 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6c42ba7d4c714a91996f66f172fa8c4d [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "6c42ba7d4c714a91996f66f172fa8c4d" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6c42ba7d4c714a91996f66f172fa8c4d" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 37905 } } }
I20260620 14:48:42.778149 10279 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6c42ba7d4c714a91996f66f172fa8c4d [sys.catalog]: This master's current role is: LEADER
I20260620 14:48:42.780299 10286 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 14:48:42.785383 10286 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 14:48:42.794696 10286 catalog_manager.cc:1347] Generated new cluster ID: ef25168c8b6540acbd6d56902dfff3b5
I20260620 14:48:42.794828 10286 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 14:48:42.805390 10286 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 14:48:42.805835 10286 catalog_manager.cc:1504] Loading token signing keys...
I20260620 14:48:42.817297 10286 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 6c42ba7d4c714a91996f66f172fa8c4d: Generated new TSK 0
I20260620 14:48:42.818212 10286 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260620 14:48:42.828570 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskUci3DU/build/asan/bin/kudu
/tmp/dist-test-taskUci3DU/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskUci3DU/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781966829091300-388-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskUci3DU/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781966829091300-388-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskUci3DU/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781966829091300-388-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskUci3DU/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781966829091300-388-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.1:0
--local_ip_for_outbound_sockets=127.0.97.1
--webserver_interface=127.0.97.1
--webserver_port=0
--tserver_master_addrs=127.0.97.62:37905
--builtin_ntp_servers=127.0.97.20:41805
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_encryption=disabled
--rpc_authentication=disabled with env {}
W20260620 14:48:43.165249 10297 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 14:48:43.165536 10297 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 14:48:43.165623 10297 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 14:48:43.172976 10297 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 14:48:43.173089 10297 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 14:48:43.173198 10297 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.1
I20260620 14:48:43.182742 10297 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:41805
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskUci3DU/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781966829091300-388-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskUci3DU/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781966829091300-388-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskUci3DU/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781966829091300-388-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.97.1
--webserver_port=0
--tserver_master_addrs=127.0.97.62:37905
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.1
--log_dir=/tmp/dist-test-taskUci3DU/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781966829091300-388-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 13:41:20 UTC on ccycloud.quasar-vdagoi.root.comops.site
build id 4457462
ASAN enabled
I20260620 14:48:43.184119 10297 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 14:48:43.185931 10297 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 14:48:43.199792 10302 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 14:48:43.199841 10305 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 14:48:43.199760 10303 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 14:48:43.200945 10297 server_base.cc:1034] running on GCE node
I20260620 14:48:43.201601 10297 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 14:48:43.202616 10297 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 14:48:43.203855 10297 hybrid_clock.cc:648] HybridClock initialized: now 1781966923203782 us; error 48 us; skew 500 ppm
I20260620 14:48:43.204198 10297 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260620 14:48:43.206537 10297 webserver.cc:460] Webserver started at http://127.0.97.1:35769/ using document root <none> and password file <none>
I20260620 14:48:43.207249 10297 fs_manager.cc:359] Metadata directory not provided
I20260620 14:48:43.207350 10297 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 14:48:43.207700 10297 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 14:48:43.213006 10297 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskUci3DU/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781966829091300-388-0/minicluster-data/ts-0/data/instance:
uuid: "69a09fb3ff2d4f958043e6f3631c0a5f"
format_stamp: "Formatted at 2026-06-20 14:48:43 on dist-test-slave-q9j6"
server_key: "d7153a2763ec5c64d7aa03b45740778c"
server_key_iv: "27b4b179bc2e487d6950e0c844999de7"
server_key_version: "encryptionkey@0"
I20260620 14:48:43.213969 10297 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskUci3DU/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781966829091300-388-0/minicluster-data/ts-0/wal/instance:
uuid: "69a09fb3ff2d4f958043e6f3631c0a5f"
format_stamp: "Formatted at 2026-06-20 14:48:43 on dist-test-slave-q9j6"
server_key: "d7153a2763ec5c64d7aa03b45740778c"
server_key_iv: "27b4b179bc2e487d6950e0c844999de7"
server_key_version: "encryptionkey@0"
I20260620 14:48:43.220366 10297 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.007s sys 0.000s
I20260620 14:48:43.224757 10311 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 14:48:43.226266 10297 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260620 14:48:43.226454 10297 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskUci3DU/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781966829091300-388-0/minicluster-data/ts-0/data,/tmp/dist-test-taskUci3DU/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781966829091300-388-0/minicluster-data/ts-0/wal
uuid: "69a09fb3ff2d4f958043e6f3631c0a5f"
format_stamp: "Formatted at 2026-06-20 14:48:43 on dist-test-slave-q9j6"
server_key: "d7153a2763ec5c64d7aa03b45740778c"
server_key_iv: "27b4b179bc2e487d6950e0c844999de7"
server_key_version: "encryptionkey@0"
I20260620 14:48:43.226713 10297 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskUci3DU/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781966829091300-388-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskUci3DU/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781966829091300-388-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskUci3DU/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781966829091300-388-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 14:48:43.232612 10297 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 14:48:43.233455 10297 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 14:48:43.233734 10297 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 14:48:43.234723 10297 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 14:48:43.236325 10297 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 14:48:43.236415 10297 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 14:48:43.236569 10297 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 14:48:43.236663 10297 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 14:48:43.269698 10297 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.1:41467
I20260620 14:48:43.269745 10423 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.1:41467 every 8 connection(s)
I20260620 14:48:43.271135 10297 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskUci3DU/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781966829091300-388-0/minicluster-data/ts-0/data/info.pb
I20260620 14:48:43.271485 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskUci3DU/build/asan/bin/kudu as pid 10297
I20260620 14:48:43.271708 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskUci3DU/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781966829091300-388-0/minicluster-data/ts-0/wal/instance
I20260620 14:48:43.272588 388 external_mini_cluster.cc:1428] Setting key fd3f100d49c6764efd80299e7d6a5da6
I20260620 14:48:43.275740 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskUci3DU/build/asan/bin/kudu
/tmp/dist-test-taskUci3DU/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskUci3DU/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781966829091300-388-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskUci3DU/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781966829091300-388-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskUci3DU/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781966829091300-388-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskUci3DU/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781966829091300-388-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.2:0
--local_ip_for_outbound_sockets=127.0.97.2
--webserver_interface=127.0.97.2
--webserver_port=0
--tserver_master_addrs=127.0.97.62:37905
--builtin_ntp_servers=127.0.97.20:41805
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_encryption=disabled
--rpc_authentication=disabled with env {}
I20260620 14:48:43.281348 10426 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:48:43.274024 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.1:42381 (local address 127.0.97.62:37905)
0620 14:48:43.274504 (+ 480us) server_negotiation.cc:206] Beginning negotiation
0620 14:48:43.274511 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0620 14:48:43.276192 (+ 1681us) server_negotiation.cc:407] Connection header received
0620 14:48:43.278872 (+ 2680us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:48:43.278877 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:48:43.278973 (+ 96us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:48:43.279070 (+ 97us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:48:43.280612 (+ 1542us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 14:48:43.280616 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 14:48:43.280619 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 14:48:43.280636 (+ 17us) server_negotiation.cc:922] calling sasl_server_start()
0620 14:48:43.280678 (+ 42us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 14:48:43.280690 (+ 12us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 14:48:43.280791 (+ 101us) server_negotiation.cc:1035] Waiting for connection context
0620 14:48:43.281028 (+ 237us) server_negotiation.cc:299] Negotiation successful
0620 14:48:43.281091 (+ 63us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":357,"thread_start_us":242,"threads_started":1}
I20260620 14:48:43.282665 10427 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:48:43.274683 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:37905 (local address 127.0.97.1:42381)
0620 14:48:43.276010 (+ 1327us) negotiation.cc:107] Waiting for socket to connect
0620 14:48:43.276069 (+ 59us) client_negotiation.cc:174] Beginning negotiation
0620 14:48:43.278562 (+ 2493us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 14:48:43.279346 (+ 784us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 14:48:43.279360 (+ 14us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 14:48:43.279504 (+ 144us) client_negotiation.cc:189] Negotiated authn=SASL
0620 14:48:43.280252 (+ 748us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0620 14:48:43.280258 (+ 6us) client_negotiation.cc:656] calling sasl_client_start()
0620 14:48:43.280313 (+ 55us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0620 14:48:43.280442 (+ 129us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 14:48:43.280790 (+ 348us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 14:48:43.280803 (+ 13us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 14:48:43.280814 (+ 11us) client_negotiation.cc:769] Sending connection context
0620 14:48:43.281109 (+ 295us) client_negotiation.cc:240] Negotiation successful
0620 14:48:43.281391 (+ 282us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":623,"thread_start_us":359,"threads_started":1}
I20260620 14:48:43.284240 10424 heartbeater.cc:347] Connected to a master server at 127.0.97.62:37905
I20260620 14:48:43.284601 10424 heartbeater.cc:464] Registering TS with master...
I20260620 14:48:43.285287 10424 heartbeater.cc:511] Master 127.0.97.62:37905 requested a full tablet report, sending...
I20260620 14:48:43.287405 10238 ts_manager.cc:194] Registered new tserver with Master: 69a09fb3ff2d4f958043e6f3631c0a5f (127.0.97.1:41467)
W20260620 14:48:43.609486 10428 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 14:48:43.609773 10428 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 14:48:43.609949 10428 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 14:48:43.617583 10428 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 14:48:43.617702 10428 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 14:48:43.617877 10428 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.2
I20260620 14:48:43.627188 10428 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:41805
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskUci3DU/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781966829091300-388-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskUci3DU/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781966829091300-388-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskUci3DU/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781966829091300-388-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.97.2
--webserver_port=0
--tserver_master_addrs=127.0.97.62:37905
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.2
--log_dir=/tmp/dist-test-taskUci3DU/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781966829091300-388-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 13:41:20 UTC on ccycloud.quasar-vdagoi.root.comops.site
build id 4457462
ASAN enabled
I20260620 14:48:43.628588 10428 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 14:48:43.630514 10428 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 14:48:43.644147 10436 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 14:48:43.644273 10433 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 14:48:43.644240 10434 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 14:48:43.645285 10428 server_base.cc:1034] running on GCE node
I20260620 14:48:43.646045 10428 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 14:48:43.647039 10428 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 14:48:43.648286 10428 hybrid_clock.cc:648] HybridClock initialized: now 1781966923648217 us; error 65 us; skew 500 ppm
I20260620 14:48:43.648633 10428 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260620 14:48:43.651034 10428 webserver.cc:460] Webserver started at http://127.0.97.2:45945/ using document root <none> and password file <none>
I20260620 14:48:43.651873 10428 fs_manager.cc:359] Metadata directory not provided
I20260620 14:48:43.652010 10428 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 14:48:43.652331 10428 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 14:48:43.655383 10428 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskUci3DU/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781966829091300-388-0/minicluster-data/ts-1/data/instance:
uuid: "9b6f992044774a8e8063053e54f491f9"
format_stamp: "Formatted at 2026-06-20 14:48:43 on dist-test-slave-q9j6"
server_key: "807b121c74c836403782ea488f4df967"
server_key_iv: "7f168ce755952bb4295b6ab620444126"
server_key_version: "encryptionkey@0"
I20260620 14:48:43.656306 10428 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskUci3DU/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781966829091300-388-0/minicluster-data/ts-1/wal/instance:
uuid: "9b6f992044774a8e8063053e54f491f9"
format_stamp: "Formatted at 2026-06-20 14:48:43 on dist-test-slave-q9j6"
server_key: "807b121c74c836403782ea488f4df967"
server_key_iv: "7f168ce755952bb4295b6ab620444126"
server_key_version: "encryptionkey@0"
I20260620 14:48:43.661804 10428 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260620 14:48:43.665560 10442 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 14:48:43.667050 10428 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260620 14:48:43.667260 10428 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskUci3DU/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781966829091300-388-0/minicluster-data/ts-1/data,/tmp/dist-test-taskUci3DU/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781966829091300-388-0/minicluster-data/ts-1/wal
uuid: "9b6f992044774a8e8063053e54f491f9"
format_stamp: "Formatted at 2026-06-20 14:48:43 on dist-test-slave-q9j6"
server_key: "807b121c74c836403782ea488f4df967"
server_key_iv: "7f168ce755952bb4295b6ab620444126"
server_key_version: "encryptionkey@0"
I20260620 14:48:43.667514 10428 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskUci3DU/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781966829091300-388-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskUci3DU/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781966829091300-388-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskUci3DU/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781966829091300-388-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 14:48:43.673607 10428 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 14:48:43.674474 10428 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 14:48:43.674757 10428 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 14:48:43.675966 10428 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 14:48:43.677446 10428 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 14:48:43.677572 10428 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 14:48:43.677716 10428 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 14:48:43.677801 10428 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 14:48:43.711475 10428 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.2:44175
I20260620 14:48:43.711568 10554 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.2:44175 every 8 connection(s)
I20260620 14:48:43.712885 10428 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskUci3DU/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781966829091300-388-0/minicluster-data/ts-1/data/info.pb
I20260620 14:48:43.721114 10426 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:48:43.715525 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.2:56545 (local address 127.0.97.62:37905)
0620 14:48:43.715768 (+ 243us) server_negotiation.cc:206] Beginning negotiation
0620 14:48:43.715775 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0620 14:48:43.717426 (+ 1651us) server_negotiation.cc:407] Connection header received
0620 14:48:43.719026 (+ 1600us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:48:43.719030 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:48:43.719095 (+ 65us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:48:43.719181 (+ 86us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:48:43.720506 (+ 1325us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 14:48:43.720510 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 14:48:43.720513 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 14:48:43.720529 (+ 16us) server_negotiation.cc:922] calling sasl_server_start()
0620 14:48:43.720570 (+ 41us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 14:48:43.720583 (+ 13us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 14:48:43.720696 (+ 113us) server_negotiation.cc:1035] Waiting for connection context
0620 14:48:43.720840 (+ 144us) server_negotiation.cc:299] Negotiation successful
0620 14:48:43.720892 (+ 52us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":104}
I20260620 14:48:43.721793 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskUci3DU/build/asan/bin/kudu as pid 10428
I20260620 14:48:43.721912 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskUci3DU/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781966829091300-388-0/minicluster-data/ts-1/wal/instance
I20260620 14:48:43.722230 10557 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:48:43.716312 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:37905 (local address 127.0.97.2:56545)
0620 14:48:43.717270 (+ 958us) negotiation.cc:107] Waiting for socket to connect
0620 14:48:43.717312 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0620 14:48:43.718781 (+ 1469us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 14:48:43.719392 (+ 611us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 14:48:43.719400 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 14:48:43.719498 (+ 98us) client_negotiation.cc:189] Negotiated authn=SASL
0620 14:48:43.720216 (+ 718us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0620 14:48:43.720218 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0620 14:48:43.720257 (+ 39us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0620 14:48:43.720341 (+ 84us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 14:48:43.720670 (+ 329us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 14:48:43.720676 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 14:48:43.720682 (+ 6us) client_negotiation.cc:769] Sending connection context
0620 14:48:43.720889 (+ 207us) client_negotiation.cc:240] Negotiation successful
0620 14:48:43.721114 (+ 225us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":471,"thread_start_us":286,"threads_started":1}
I20260620 14:48:43.722446 388 external_mini_cluster.cc:1428] Setting key aa5138365ee21c6a1da8c062a567d34d
I20260620 14:48:43.723713 10555 heartbeater.cc:347] Connected to a master server at 127.0.97.62:37905
I20260620 14:48:43.724062 10555 heartbeater.cc:464] Registering TS with master...
I20260620 14:48:43.724745 10555 heartbeater.cc:511] Master 127.0.97.62:37905 requested a full tablet report, sending...
I20260620 14:48:43.725339 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskUci3DU/build/asan/bin/kudu
/tmp/dist-test-taskUci3DU/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskUci3DU/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781966829091300-388-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskUci3DU/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781966829091300-388-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskUci3DU/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781966829091300-388-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskUci3DU/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781966829091300-388-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.3:0
--local_ip_for_outbound_sockets=127.0.97.3
--webserver_interface=127.0.97.3
--webserver_port=0
--tserver_master_addrs=127.0.97.62:37905
--builtin_ntp_servers=127.0.97.20:41805
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_encryption=disabled
--rpc_authentication=disabled with env {}
I20260620 14:48:43.726150 10238 ts_manager.cc:194] Registered new tserver with Master: 9b6f992044774a8e8063053e54f491f9 (127.0.97.2:44175)
W20260620 14:48:44.065963 10558 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 14:48:44.066278 10558 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 14:48:44.066368 10558 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 14:48:44.073849 10558 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 14:48:44.073961 10558 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 14:48:44.074069 10558 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.3
I20260620 14:48:44.083362 10558 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:41805
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskUci3DU/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781966829091300-388-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskUci3DU/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781966829091300-388-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskUci3DU/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781966829091300-388-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.97.3
--webserver_port=0
--tserver_master_addrs=127.0.97.62:37905
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.3
--log_dir=/tmp/dist-test-taskUci3DU/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781966829091300-388-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 13:41:20 UTC on ccycloud.quasar-vdagoi.root.comops.site
build id 4457462
ASAN enabled
I20260620 14:48:44.084722 10558 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 14:48:44.086450 10558 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 14:48:44.099747 10563 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 14:48:44.099701 10564 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 14:48:44.099785 10566 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 14:48:44.100785 10558 server_base.cc:1034] running on GCE node
I20260620 14:48:44.101418 10558 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 14:48:44.102409 10558 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 14:48:44.103701 10558 hybrid_clock.cc:648] HybridClock initialized: now 1781966924103624 us; error 67 us; skew 500 ppm
I20260620 14:48:44.104023 10558 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260620 14:48:44.106400 10558 webserver.cc:460] Webserver started at http://127.0.97.3:46865/ using document root <none> and password file <none>
I20260620 14:48:44.107129 10558 fs_manager.cc:359] Metadata directory not provided
I20260620 14:48:44.107263 10558 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 14:48:44.107589 10558 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 14:48:44.110616 10558 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskUci3DU/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781966829091300-388-0/minicluster-data/ts-2/data/instance:
uuid: "5eb59828b68141a79f480eed6d2b6652"
format_stamp: "Formatted at 2026-06-20 14:48:44 on dist-test-slave-q9j6"
server_key: "11992c728f865e4f715f0fdb466d3cc6"
server_key_iv: "6fa703b48d2dc537e646dbbf815b5b17"
server_key_version: "encryptionkey@0"
I20260620 14:48:44.111451 10558 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskUci3DU/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781966829091300-388-0/minicluster-data/ts-2/wal/instance:
uuid: "5eb59828b68141a79f480eed6d2b6652"
format_stamp: "Formatted at 2026-06-20 14:48:44 on dist-test-slave-q9j6"
server_key: "11992c728f865e4f715f0fdb466d3cc6"
server_key_iv: "6fa703b48d2dc537e646dbbf815b5b17"
server_key_version: "encryptionkey@0"
I20260620 14:48:44.116926 10558 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.004s sys 0.000s
I20260620 14:48:44.120494 10572 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 14:48:44.121944 10558 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.003s
I20260620 14:48:44.122150 10558 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskUci3DU/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781966829091300-388-0/minicluster-data/ts-2/data,/tmp/dist-test-taskUci3DU/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781966829091300-388-0/minicluster-data/ts-2/wal
uuid: "5eb59828b68141a79f480eed6d2b6652"
format_stamp: "Formatted at 2026-06-20 14:48:44 on dist-test-slave-q9j6"
server_key: "11992c728f865e4f715f0fdb466d3cc6"
server_key_iv: "6fa703b48d2dc537e646dbbf815b5b17"
server_key_version: "encryptionkey@0"
I20260620 14:48:44.122409 10558 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskUci3DU/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781966829091300-388-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskUci3DU/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781966829091300-388-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskUci3DU/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781966829091300-388-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 14:48:44.128396 10558 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 14:48:44.129249 10558 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 14:48:44.129525 10558 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 14:48:44.130535 10558 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 14:48:44.132432 10558 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 14:48:44.132565 10558 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 14:48:44.132706 10558 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 14:48:44.132793 10558 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 14:48:44.165894 10558 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.3:36293
I20260620 14:48:44.165943 10684 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.3:36293 every 8 connection(s)
I20260620 14:48:44.167269 10558 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskUci3DU/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781966829091300-388-0/minicluster-data/ts-2/data/info.pb
I20260620 14:48:44.169180 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskUci3DU/build/asan/bin/kudu as pid 10558
I20260620 14:48:44.169354 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskUci3DU/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781966829091300-388-0/minicluster-data/ts-2/wal/instance
I20260620 14:48:44.169817 388 external_mini_cluster.cc:1428] Setting key 3bb30658a5ac74655b7525f16c4716ec
I20260620 14:48:44.176430 10426 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:48:44.170427 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.3:36211 (local address 127.0.97.62:37905)
0620 14:48:44.170620 (+ 193us) server_negotiation.cc:206] Beginning negotiation
0620 14:48:44.170628 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0620 14:48:44.172069 (+ 1441us) server_negotiation.cc:407] Connection header received
0620 14:48:44.174044 (+ 1975us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:48:44.174049 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:48:44.174118 (+ 69us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:48:44.174239 (+ 121us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:48:44.175675 (+ 1436us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 14:48:44.175680 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 14:48:44.175685 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 14:48:44.175704 (+ 19us) server_negotiation.cc:922] calling sasl_server_start()
0620 14:48:44.175745 (+ 41us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 14:48:44.175757 (+ 12us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 14:48:44.175880 (+ 123us) server_negotiation.cc:1035] Waiting for connection context
0620 14:48:44.176089 (+ 209us) server_negotiation.cc:299] Negotiation successful
0620 14:48:44.176160 (+ 71us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":73}
I20260620 14:48:44.177440 10687 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:48:44.170641 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:37905 (local address 127.0.97.3:36211)
0620 14:48:44.171925 (+ 1284us) negotiation.cc:107] Waiting for socket to connect
0620 14:48:44.171981 (+ 56us) client_negotiation.cc:174] Beginning negotiation
0620 14:48:44.173778 (+ 1797us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 14:48:44.174479 (+ 701us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 14:48:44.174488 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 14:48:44.174592 (+ 104us) client_negotiation.cc:189] Negotiated authn=SASL
0620 14:48:44.175312 (+ 720us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0620 14:48:44.175314 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0620 14:48:44.175355 (+ 41us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0620 14:48:44.175490 (+ 135us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 14:48:44.175851 (+ 361us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 14:48:44.175861 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 14:48:44.175869 (+ 8us) client_negotiation.cc:769] Sending connection context
0620 14:48:44.176173 (+ 304us) client_negotiation.cc:240] Negotiation successful
0620 14:48:44.176450 (+ 277us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":615,"thread_start_us":404,"threads_started":1}
I20260620 14:48:44.178952 10685 heartbeater.cc:347] Connected to a master server at 127.0.97.62:37905
I20260620 14:48:44.179284 10685 heartbeater.cc:464] Registering TS with master...
I20260620 14:48:44.179986 10685 heartbeater.cc:511] Master 127.0.97.62:37905 requested a full tablet report, sending...
I20260620 14:48:44.181500 10238 ts_manager.cc:194] Registered new tserver with Master: 5eb59828b68141a79f480eed6d2b6652 (127.0.97.3:36293)
I20260620 14:48:44.185478 388 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
W20260620 14:48:44.191378 10426 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 14:48:44.189868 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37624 (local address 127.0.97.62:37905)
0620 14:48:44.190075 (+ 207us) server_negotiation.cc:206] Beginning negotiation
0620 14:48:44.190084 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0620 14:48:44.190534 (+ 450us) server_negotiation.cc:407] Connection header received
0620 14:48:44.190674 (+ 140us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:48:44.190678 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:48:44.190731 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:48:44.190815 (+ 84us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:48:44.191149 (+ 334us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:37624: BlockingRecv error: recv got EOF from 127.0.0.1:37624 (error 108)
Metrics: {"server-negotiator.queue_time_us":61}
I20260620 14:48:44.191802 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskUci3DU/build/asan/bin/kudu pid 10297
I20260620 14:48:44.195242 10694 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:48:44.192800 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40086 (local address 127.0.97.1:41467)
0620 14:48:44.193329 (+ 529us) server_negotiation.cc:206] Beginning negotiation
0620 14:48:44.193342 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0620 14:48:44.193366 (+ 24us) server_negotiation.cc:407] Connection header received
0620 14:48:44.193422 (+ 56us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:48:44.193429 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:48:44.193639 (+ 210us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:48:44.193765 (+ 126us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:48:44.194630 (+ 865us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 14:48:44.194636 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 14:48:44.194647 (+ 11us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 14:48:44.194662 (+ 15us) server_negotiation.cc:922] calling sasl_server_start()
0620 14:48:44.194698 (+ 36us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 14:48:44.194712 (+ 14us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 14:48:44.194825 (+ 113us) server_negotiation.cc:1035] Waiting for connection context
0620 14:48:44.194918 (+ 93us) server_negotiation.cc:299] Negotiation successful
0620 14:48:44.195010 (+ 92us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":379,"thread_start_us":266,"threads_started":1}
I20260620 14:48:44.196606 10419 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 14:48:44.280086 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskUci3DU/build/asan/bin/kudu with pid 10297
I20260620 14:48:44.296706 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskUci3DU/build/asan/bin/kudu pid 10428
I20260620 14:48:44.300315 10696 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:48:44.297705 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50114 (local address 127.0.97.2:44175)
0620 14:48:44.298201 (+ 496us) server_negotiation.cc:206] Beginning negotiation
0620 14:48:44.298208 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0620 14:48:44.298234 (+ 26us) server_negotiation.cc:407] Connection header received
0620 14:48:44.298295 (+ 61us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:48:44.298301 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:48:44.298527 (+ 226us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:48:44.298653 (+ 126us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:48:44.299539 (+ 886us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 14:48:44.299546 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 14:48:44.299559 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 14:48:44.299578 (+ 19us) server_negotiation.cc:922] calling sasl_server_start()
0620 14:48:44.299686 (+ 108us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 14:48:44.299704 (+ 18us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 14:48:44.299811 (+ 107us) server_negotiation.cc:1035] Waiting for connection context
0620 14:48:44.299898 (+ 87us) server_negotiation.cc:299] Negotiation successful
0620 14:48:44.300013 (+ 115us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":370,"thread_start_us":244,"threads_started":1}
I20260620 14:48:44.301537 10550 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 14:48:44.382736 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskUci3DU/build/asan/bin/kudu with pid 10428
I20260620 14:48:44.398939 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskUci3DU/build/asan/bin/kudu pid 10558
I20260620 14:48:44.403014 10698 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:48:44.399998 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38724 (local address 127.0.97.3:36293)
0620 14:48:44.400606 (+ 608us) server_negotiation.cc:206] Beginning negotiation
0620 14:48:44.400613 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0620 14:48:44.400643 (+ 30us) server_negotiation.cc:407] Connection header received
0620 14:48:44.400701 (+ 58us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:48:44.400708 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:48:44.400926 (+ 218us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:48:44.401112 (+ 186us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:48:44.402264 (+ 1152us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 14:48:44.402270 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 14:48:44.402283 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 14:48:44.402299 (+ 16us) server_negotiation.cc:922] calling sasl_server_start()
0620 14:48:44.402337 (+ 38us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 14:48:44.402350 (+ 13us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 14:48:44.402538 (+ 188us) server_negotiation.cc:1035] Waiting for connection context
0620 14:48:44.402612 (+ 74us) server_negotiation.cc:299] Negotiation successful
0620 14:48:44.402717 (+ 105us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":467,"thread_start_us":310,"threads_started":1}
I20260620 14:48:44.404340 10680 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 14:48:44.486196 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskUci3DU/build/asan/bin/kudu with pid 10558
I20260620 14:48:44.501858 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskUci3DU/build/asan/bin/kudu pid 10207
I20260620 14:48:44.503078 10268 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 14:48:44.603143 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskUci3DU/build/asan/bin/kudu with pid 10207
2026-06-20T14:48:44Z chronyd exiting
[ OK ] SecurityITest.TestRequireEncryptionInsecureCluster (2362 ms)
[ RUN ] SecurityITest.TestRequireAuthenticationSecureCluster
Loading random data
Initializing database '/tmp/dist-test-taskUci3DU/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781966829091300-388-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 20 14:48:44 dist-test-slave-q9j6 krb5kdc[10703](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 20 14:48:44 dist-test-slave-q9j6 krb5kdc[10703](info): set up 2 sockets
Jun 20 14:48:44 dist-test-slave-q9j6 krb5kdc[10703](info): commencing operation
krb5kdc: starting...
W20260620 14:48:45.708645 388 mini_kdc.cc:121] Time spent starting KDC: real 1.061s user 0.003s sys 0.020s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskUci3DU/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781966829091300-388-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskUci3DU/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781966829091300-388-0/krb5kdc/test-user.keytab.
Jun 20 14:48:45 dist-test-slave-q9j6 krb5kdc[10703](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781966925, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-20T14:48:45Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T14:48:45Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.97.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskUci3DU/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781966829091300-388-0/krb5kdc/kudu_127.0.97.62.keytab.
Entry for principal kudu/127.0.97.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskUci3DU/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781966829091300-388-0/krb5kdc/kudu_127.0.97.62.keytab.
I20260620 14:48:45.961076 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskUci3DU/build/asan/bin/kudu
/tmp/dist-test-taskUci3DU/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskUci3DU/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781966829091300-388-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskUci3DU/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781966829091300-388-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskUci3DU/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781966829091300-388-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskUci3DU/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781966829091300-388-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:41487
--webserver_interface=127.0.97.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskUci3DU/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781966829091300-388-0/krb5kdc/kudu_127.0.97.62.keytab
--principal=kudu/127.0.97.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:34299
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.97.62:41487
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskUci3DU/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781966829091300-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskUci3DU/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781966829091300-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskUci3DU/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781966829091300-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 14:48:46.295130 10716 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 14:48:46.295425 10716 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 14:48:46.295514 10716 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 14:48:46.302892 10716 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 14:48:46.303023 10716 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 14:48:46.303069 10716 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 14:48:46.303125 10716 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 14:48:46.303167 10716 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 14:48:46.313009 10716 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:34299
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskUci3DU/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781966829091300-388-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskUci3DU/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781966829091300-388-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.97.62:41487
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskUci3DU/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781966829091300-388-0/krb5kdc/kudu_127.0.97.62.keytab
--principal=kudu/127.0.97.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:41487
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskUci3DU/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781966829091300-388-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.97.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskUci3DU/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781966829091300-388-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 13:41:20 UTC on ccycloud.quasar-vdagoi.root.comops.site
build id 4457462
ASAN enabled
I20260620 14:48:46.314294 10716 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 14:48:46.315521 10716 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 14:48:46.327036 10721 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 14:48:46.327082 10722 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 14:48:46.327118 10724 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 14:48:46.328258 10716 server_base.cc:1034] running on GCE node
I20260620 14:48:46.329061 10716 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 14:48:46.330412 10716 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 14:48:46.331683 10716 hybrid_clock.cc:648] HybridClock initialized: now 1781966926331640 us; error 56 us; skew 500 ppm
I20260620 14:48:46.332039 10716 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 14:48:46 dist-test-slave-q9j6 krb5kdc[10703](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781966926, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 14:48:46.334964 10716 init.cc:377] Logged in from keytab as kudu/127.0.97.62@KRBTEST.COM (short username kudu)
I20260620 14:48:46.336483 10716 webserver.cc:460] Webserver started at http://127.0.97.62:34489/ using document root <none> and password file <none>
I20260620 14:48:46.337194 10716 fs_manager.cc:359] Metadata directory not provided
I20260620 14:48:46.337337 10716 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 14:48:46.337641 10716 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 14:48:46.340729 10716 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskUci3DU/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781966829091300-388-0/minicluster-data/master-0/data/instance:
uuid: "e87c3c7c892b42da8845070c0e2d98f8"
format_stamp: "Formatted at 2026-06-20 14:48:46 on dist-test-slave-q9j6"
server_key: "a8f5fee23d5d13200b2f4cf865145d6a"
server_key_iv: "23c2907c8bb53d7868cfec9c1a20bfe0"
server_key_version: "encryptionkey@0"
I20260620 14:48:46.341544 10716 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskUci3DU/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781966829091300-388-0/minicluster-data/master-0/wal/instance:
uuid: "e87c3c7c892b42da8845070c0e2d98f8"
format_stamp: "Formatted at 2026-06-20 14:48:46 on dist-test-slave-q9j6"
server_key: "a8f5fee23d5d13200b2f4cf865145d6a"
server_key_iv: "23c2907c8bb53d7868cfec9c1a20bfe0"
server_key_version: "encryptionkey@0"
I20260620 14:48:46.347219 10716 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.000s
I20260620 14:48:46.350823 10731 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 14:48:46.352308 10716 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.002s
I20260620 14:48:46.352501 10716 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskUci3DU/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781966829091300-388-0/minicluster-data/master-0/data,/tmp/dist-test-taskUci3DU/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781966829091300-388-0/minicluster-data/master-0/wal
uuid: "e87c3c7c892b42da8845070c0e2d98f8"
format_stamp: "Formatted at 2026-06-20 14:48:46 on dist-test-slave-q9j6"
server_key: "a8f5fee23d5d13200b2f4cf865145d6a"
server_key_iv: "23c2907c8bb53d7868cfec9c1a20bfe0"
server_key_version: "encryptionkey@0"
I20260620 14:48:46.352710 10716 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskUci3DU/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781966829091300-388-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskUci3DU/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781966829091300-388-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskUci3DU/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781966829091300-388-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 14:48:46.390256 10716 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 14:48:46.394564 10716 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 14:48:46.394856 10716 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 14:48:46.413453 10716 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.62:41487
I20260620 14:48:46.413484 10782 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.62:41487 every 8 connection(s)
I20260620 14:48:46.414916 10716 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskUci3DU/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781966829091300-388-0/minicluster-data/master-0/data/info.pb
I20260620 14:48:46.418776 10783 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 14:48:46.424896 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskUci3DU/build/asan/bin/kudu as pid 10716
I20260620 14:48:46.425122 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskUci3DU/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781966829091300-388-0/minicluster-data/master-0/wal/instance
I20260620 14:48:46.425901 388 external_mini_cluster.cc:1428] Setting key 82dfd4c81777390a210566d24f3e7740
I20260620 14:48:46.430361 10783 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e87c3c7c892b42da8845070c0e2d98f8: Bootstrap starting.
I20260620 14:48:46.434162 10783 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P e87c3c7c892b42da8845070c0e2d98f8: Neither blocks nor log segments found. Creating new log.
I20260620 14:48:46.435417 10783 log.cc:824] T 00000000000000000000000000000000 P e87c3c7c892b42da8845070c0e2d98f8: Log is configured to *not* fsync() on all Append() calls
I20260620 14:48:46.438812 10783 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e87c3c7c892b42da8845070c0e2d98f8: No bootstrap required, opened a new log
Jun 20 14:48:46 dist-test-slave-q9j6 krb5kdc[10703](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781966925, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260620 14:48:46.447031 10783 raft_consensus.cc:348] T 00000000000000000000000000000000 P e87c3c7c892b42da8845070c0e2d98f8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e87c3c7c892b42da8845070c0e2d98f8" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 41487 } }
I20260620 14:48:46.447431 10783 raft_consensus.cc:374] T 00000000000000000000000000000000 P e87c3c7c892b42da8845070c0e2d98f8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 14:48:46.447547 10783 raft_consensus.cc:729] T 00000000000000000000000000000000 P e87c3c7c892b42da8845070c0e2d98f8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e87c3c7c892b42da8845070c0e2d98f8, State: Initialized, Role: FOLLOWER
I20260620 14:48:46.448308 10783 consensus_queue.cc:260] T 00000000000000000000000000000000 P e87c3c7c892b42da8845070c0e2d98f8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e87c3c7c892b42da8845070c0e2d98f8" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 41487 } }
I20260620 14:48:46.448563 10783 raft_consensus.cc:388] T 00000000000000000000000000000000 P e87c3c7c892b42da8845070c0e2d98f8 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 14:48:46.448722 10783 raft_consensus.cc:482] T 00000000000000000000000000000000 P e87c3c7c892b42da8845070c0e2d98f8 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 14:48:46.448927 10783 raft_consensus.cc:3037] T 00000000000000000000000000000000 P e87c3c7c892b42da8845070c0e2d98f8 [term 0 FOLLOWER]: Advancing to term 1
I20260620 14:48:46.449618 10786 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:48:46.428084 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36390 (local address 127.0.97.62:41487)
0620 14:48:46.428908 (+ 824us) server_negotiation.cc:206] Beginning negotiation
0620 14:48:46.428921 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0620 14:48:46.428961 (+ 40us) server_negotiation.cc:407] Connection header received
0620 14:48:46.430660 (+ 1699us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:48:46.430684 (+ 24us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:48:46.431093 (+ 409us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:48:46.431427 (+ 334us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:48:46.433348 (+ 1921us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:48:46.434746 (+ 1398us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:48:46.436587 (+ 1841us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:48:46.437369 (+ 782us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:48:46.437585 (+ 216us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:48:46.441693 (+ 4108us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 14:48:46.441702 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 14:48:46.441724 (+ 22us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 14:48:46.441764 (+ 40us) server_negotiation.cc:922] calling sasl_server_start()
0620 14:48:46.445045 (+ 3281us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:48:46.445852 (+ 807us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:48:46.445862 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:48:46.445873 (+ 11us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:48:46.446054 (+ 181us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:48:46.446471 (+ 417us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:48:46.446480 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:48:46.446482 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:48:46.447075 (+ 593us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 14:48:46.447389 (+ 314us) server_negotiation.cc:1035] Waiting for connection context
0620 14:48:46.447862 (+ 473us) server_negotiation.cc:299] Negotiation successful
0620 14:48:46.448293 (+ 431us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":615,"thread_start_us":412,"threads_started":1}
I20260620 14:48:46.451999 10783 raft_consensus.cc:504] T 00000000000000000000000000000000 P e87c3c7c892b42da8845070c0e2d98f8 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e87c3c7c892b42da8845070c0e2d98f8" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 41487 } }
I20260620 14:48:46.452592 10783 leader_election.cc:302] T 00000000000000000000000000000000 P e87c3c7c892b42da8845070c0e2d98f8 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: e87c3c7c892b42da8845070c0e2d98f8; no voters:
I20260620 14:48:46.453230 10783 leader_election.cc:288] T 00000000000000000000000000000000 P e87c3c7c892b42da8845070c0e2d98f8 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260620 14:48:46.453370 10788 raft_consensus.cc:2781] T 00000000000000000000000000000000 P e87c3c7c892b42da8845070c0e2d98f8 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 14:48:46.454070 10788 raft_consensus.cc:686] T 00000000000000000000000000000000 P e87c3c7c892b42da8845070c0e2d98f8 [term 1 LEADER]: Becoming Leader. State: Replica: e87c3c7c892b42da8845070c0e2d98f8, State: Running, Role: LEADER
I20260620 14:48:46.454612 10788 consensus_queue.cc:237] T 00000000000000000000000000000000 P e87c3c7c892b42da8845070c0e2d98f8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e87c3c7c892b42da8845070c0e2d98f8" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 41487 } }
I20260620 14:48:46.455349 10783 sys_catalog.cc:564] T 00000000000000000000000000000000 P e87c3c7c892b42da8845070c0e2d98f8 [sys.catalog]: configured and running, proceeding with master startup.
I20260620 14:48:46.457340 10789 sys_catalog.cc:455] T 00000000000000000000000000000000 P e87c3c7c892b42da8845070c0e2d98f8 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "e87c3c7c892b42da8845070c0e2d98f8" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e87c3c7c892b42da8845070c0e2d98f8" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 41487 } } }
I20260620 14:48:46.457721 10789 sys_catalog.cc:458] T 00000000000000000000000000000000 P e87c3c7c892b42da8845070c0e2d98f8 [sys.catalog]: This master's current role is: LEADER
I20260620 14:48:46.458154 10790 sys_catalog.cc:455] T 00000000000000000000000000000000 P e87c3c7c892b42da8845070c0e2d98f8 [sys.catalog]: SysCatalogTable state changed. Reason: New leader e87c3c7c892b42da8845070c0e2d98f8. Latest consensus state: current_term: 1 leader_uuid: "e87c3c7c892b42da8845070c0e2d98f8" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e87c3c7c892b42da8845070c0e2d98f8" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 41487 } } }
I20260620 14:48:46.458782 10790 sys_catalog.cc:458] T 00000000000000000000000000000000 P e87c3c7c892b42da8845070c0e2d98f8 [sys.catalog]: This master's current role is: LEADER
I20260620 14:48:46.462265 10793 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 14:48:46.467819 10793 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 14:48:46.477195 10793 catalog_manager.cc:1347] Generated new cluster ID: 1159fbd4c2534c6cb87a345681848386
I20260620 14:48:46.477319 10793 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 14:48:46.499848 10793 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 14:48:46.501235 10793 catalog_manager.cc:1504] Loading token signing keys...
I20260620 14:48:46.511209 10793 catalog_manager.cc:5868] T 00000000000000000000000000000000 P e87c3c7c892b42da8845070c0e2d98f8: Generated new TSK 0
I20260620 14:48:46.512163 10793 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.0.97.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskUci3DU/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781966829091300-388-0/krb5kdc/kudu_127.0.97.1.keytab.
Entry for principal kudu/127.0.97.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskUci3DU/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781966829091300-388-0/krb5kdc/kudu_127.0.97.1.keytab.
I20260620 14:48:46.572427 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskUci3DU/build/asan/bin/kudu
/tmp/dist-test-taskUci3DU/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskUci3DU/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781966829091300-388-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskUci3DU/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781966829091300-388-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskUci3DU/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781966829091300-388-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskUci3DU/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781966829091300-388-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.1:0
--local_ip_for_outbound_sockets=127.0.97.1
--webserver_interface=127.0.97.1
--webserver_port=0
--tserver_master_addrs=127.0.97.62:41487
--keytab_file=/tmp/dist-test-taskUci3DU/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781966829091300-388-0/krb5kdc/kudu_127.0.97.1.keytab
--principal=kudu/127.0.97.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:34299
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskUci3DU/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781966829091300-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskUci3DU/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781966829091300-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskUci3DU/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781966829091300-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 14:48:46.914942 10810 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 14:48:46.915319 10810 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 14:48:46.915532 10810 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 14:48:46.922884 10810 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 14:48:46.923076 10810 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 14:48:46.923266 10810 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.1
I20260620 14:48:46.933254 10810 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:34299
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskUci3DU/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781966829091300-388-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskUci3DU/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781966829091300-388-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskUci3DU/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781966829091300-388-0/krb5kdc/kudu_127.0.97.1.keytab
--principal=kudu/127.0.97.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskUci3DU/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781966829091300-388-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.97.1
--webserver_port=0
--tserver_master_addrs=127.0.97.62:41487
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.1
--log_dir=/tmp/dist-test-taskUci3DU/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781966829091300-388-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 13:41:20 UTC on ccycloud.quasar-vdagoi.root.comops.site
build id 4457462
ASAN enabled
I20260620 14:48:46.934679 10810 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 14:48:46.936151 10810 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 14:48:46.949626 10815 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 14:48:46.949648 10818 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 14:48:46.949870 10816 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 14:48:46.951061 10810 server_base.cc:1034] running on GCE node
I20260620 14:48:46.951722 10810 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 14:48:46.952703 10810 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 14:48:46.953907 10810 hybrid_clock.cc:648] HybridClock initialized: now 1781966926953857 us; error 47 us; skew 500 ppm
I20260620 14:48:46.954229 10810 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 14:48:46 dist-test-slave-q9j6 krb5kdc[10703](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781966926, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 14:48:46.957726 10810 init.cc:377] Logged in from keytab as kudu/127.0.97.1@KRBTEST.COM (short username kudu)
I20260620 14:48:46.959242 10810 webserver.cc:460] Webserver started at http://127.0.97.1:46751/ using document root <none> and password file <none>
I20260620 14:48:46.960115 10810 fs_manager.cc:359] Metadata directory not provided
I20260620 14:48:46.960255 10810 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 14:48:46.960615 10810 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 14:48:46.963814 10810 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskUci3DU/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781966829091300-388-0/minicluster-data/ts-0/data/instance:
uuid: "380b5d5555c244899f8a06998284b5a0"
format_stamp: "Formatted at 2026-06-20 14:48:46 on dist-test-slave-q9j6"
server_key: "5cc390c7e82c1f05faf6c248092b4d8c"
server_key_iv: "389e5f71cf883d2b9fc2fc57ca91e380"
server_key_version: "encryptionkey@0"
I20260620 14:48:46.964563 10810 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskUci3DU/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781966829091300-388-0/minicluster-data/ts-0/wal/instance:
uuid: "380b5d5555c244899f8a06998284b5a0"
format_stamp: "Formatted at 2026-06-20 14:48:46 on dist-test-slave-q9j6"
server_key: "5cc390c7e82c1f05faf6c248092b4d8c"
server_key_iv: "389e5f71cf883d2b9fc2fc57ca91e380"
server_key_version: "encryptionkey@0"
I20260620 14:48:46.970093 10810 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260620 14:48:46.973644 10825 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 14:48:46.975032 10810 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260620 14:48:46.975168 10810 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskUci3DU/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781966829091300-388-0/minicluster-data/ts-0/data,/tmp/dist-test-taskUci3DU/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781966829091300-388-0/minicluster-data/ts-0/wal
uuid: "380b5d5555c244899f8a06998284b5a0"
format_stamp: "Formatted at 2026-06-20 14:48:46 on dist-test-slave-q9j6"
server_key: "5cc390c7e82c1f05faf6c248092b4d8c"
server_key_iv: "389e5f71cf883d2b9fc2fc57ca91e380"
server_key_version: "encryptionkey@0"
I20260620 14:48:46.975349 10810 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskUci3DU/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781966829091300-388-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskUci3DU/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781966829091300-388-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskUci3DU/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781966829091300-388-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 14:48:46.995033 10810 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 14:48:46.999049 10810 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 14:48:46.999284 10810 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 14:48:47.000258 10810 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 14:48:47.001560 10810 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 14:48:47.001631 10810 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 14:48:47.001714 10810 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 14:48:47.001757 10810 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 14:48:47.035651 10810 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.1:45285
I20260620 14:48:47.035758 10937 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.1:45285 every 8 connection(s)
I20260620 14:48:47.037071 10810 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskUci3DU/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781966829091300-388-0/minicluster-data/ts-0/data/info.pb
I20260620 14:48:47.038025 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskUci3DU/build/asan/bin/kudu as pid 10810
I20260620 14:48:47.038249 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskUci3DU/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781966829091300-388-0/minicluster-data/ts-0/wal/instance
I20260620 14:48:47.039037 388 external_mini_cluster.cc:1428] Setting key 76e9baedc206352fd0dce862230167a6
Jun 20 14:48:47 dist-test-slave-q9j6 krb5kdc[10703](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781966926, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.1@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260620 14:48:47.063228 10940 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:48:47.040243 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.1:59611 (local address 127.0.97.62:41487)
0620 14:48:47.040703 (+ 460us) server_negotiation.cc:206] Beginning negotiation
0620 14:48:47.040708 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 14:48:47.042042 (+ 1334us) server_negotiation.cc:407] Connection header received
0620 14:48:47.045017 (+ 2975us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:48:47.045022 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:48:47.045093 (+ 71us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:48:47.045193 (+ 100us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:48:47.047422 (+ 2229us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:48:47.048479 (+ 1057us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:48:47.050720 (+ 2241us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:48:47.051453 (+ 733us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:48:47.051644 (+ 191us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:48:47.055620 (+ 3976us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 14:48:47.055631 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 14:48:47.055639 (+ 8us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 14:48:47.055688 (+ 49us) server_negotiation.cc:922] calling sasl_server_start()
0620 14:48:47.059296 (+ 3608us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:48:47.060246 (+ 950us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:48:47.060255 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:48:47.060262 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:48:47.060418 (+ 156us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:48:47.060849 (+ 431us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:48:47.060856 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:48:47.060862 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:48:47.061254 (+ 392us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 14:48:47.061427 (+ 173us) server_negotiation.cc:1035] Waiting for connection context
0620 14:48:47.062266 (+ 839us) server_negotiation.cc:299] Negotiation successful
0620 14:48:47.062621 (+ 355us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":358,"thread_start_us":250,"threads_started":1}
I20260620 14:48:47.063858 10941 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:48:47.040852 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:41487 (local address 127.0.97.1:59611)
0620 14:48:47.041839 (+ 987us) negotiation.cc:107] Waiting for socket to connect
0620 14:48:47.041898 (+ 59us) client_negotiation.cc:174] Beginning negotiation
0620 14:48:47.044674 (+ 2776us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 14:48:47.045512 (+ 838us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 14:48:47.045527 (+ 15us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 14:48:47.046343 (+ 816us) client_negotiation.cc:189] Negotiated authn=SASL
0620 14:48:47.047205 (+ 862us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:48:47.047236 (+ 31us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:48:47.048665 (+ 1429us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:48:47.048674 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:48:47.050531 (+ 1857us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:48:47.050546 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:48:47.051755 (+ 1209us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:48:47.051761 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:48:47.052114 (+ 353us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:48:47.052960 (+ 846us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 14:48:47.052988 (+ 28us) client_negotiation.cc:656] calling sasl_client_start()
0620 14:48:47.055295 (+ 2307us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 14:48:47.059467 (+ 4172us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 14:48:47.059475 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 14:48:47.059492 (+ 17us) client_negotiation.cc:760] calling sasl_client_step()
0620 14:48:47.060114 (+ 622us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 14:48:47.060532 (+ 418us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 14:48:47.060535 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 14:48:47.060538 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0620 14:48:47.060719 (+ 181us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 14:48:47.061408 (+ 689us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 14:48:47.061419 (+ 11us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 14:48:47.061843 (+ 424us) client_negotiation.cc:769] Sending connection context
0620 14:48:47.062099 (+ 256us) client_negotiation.cc:240] Negotiation successful
0620 14:48:47.062513 (+ 414us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":663,"thread_start_us":368,"threads_started":1}
I20260620 14:48:47.065538 10938 heartbeater.cc:347] Connected to a master server at 127.0.97.62:41487
WARNING: no policy specified for kudu/127.0.97.2@KRBTEST.COM; defaulting to no policy
I20260620 14:48:47.065866 10938 heartbeater.cc:464] Registering TS with master...
I20260620 14:48:47.066623 10938 heartbeater.cc:511] Master 127.0.97.62:41487 requested a full tablet report, sending...
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.2@KRBTEST.COM" created.
I20260620 14:48:47.069098 10748 ts_manager.cc:194] Registered new tserver with Master: 380b5d5555c244899f8a06998284b5a0 (127.0.97.1:45285)
I20260620 14:48:47.071331 10748 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.97.1@KRBTEST.COM'} at 127.0.97.1:59611
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskUci3DU/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781966829091300-388-0/krb5kdc/kudu_127.0.97.2.keytab.
Entry for principal kudu/127.0.97.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskUci3DU/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781966829091300-388-0/krb5kdc/kudu_127.0.97.2.keytab.
I20260620 14:48:47.096405 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskUci3DU/build/asan/bin/kudu
/tmp/dist-test-taskUci3DU/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskUci3DU/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781966829091300-388-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskUci3DU/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781966829091300-388-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskUci3DU/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781966829091300-388-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskUci3DU/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781966829091300-388-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.2:0
--local_ip_for_outbound_sockets=127.0.97.2
--webserver_interface=127.0.97.2
--webserver_port=0
--tserver_master_addrs=127.0.97.62:41487
--keytab_file=/tmp/dist-test-taskUci3DU/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781966829091300-388-0/krb5kdc/kudu_127.0.97.2.keytab
--principal=kudu/127.0.97.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:34299
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskUci3DU/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781966829091300-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskUci3DU/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781966829091300-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskUci3DU/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781966829091300-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 14:48:47.436498 10944 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 14:48:47.436852 10944 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 14:48:47.437024 10944 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 14:48:47.444571 10944 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 14:48:47.444731 10944 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 14:48:47.444914 10944 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.2
I20260620 14:48:47.455534 10944 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:34299
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskUci3DU/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781966829091300-388-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskUci3DU/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781966829091300-388-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskUci3DU/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781966829091300-388-0/krb5kdc/kudu_127.0.97.2.keytab
--principal=kudu/127.0.97.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskUci3DU/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781966829091300-388-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.97.2
--webserver_port=0
--tserver_master_addrs=127.0.97.62:41487
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.2
--log_dir=/tmp/dist-test-taskUci3DU/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781966829091300-388-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 13:41:20 UTC on ccycloud.quasar-vdagoi.root.comops.site
build id 4457462
ASAN enabled
I20260620 14:48:47.457067 10944 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 14:48:47.458443 10944 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 14:48:47.472240 10949 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 14:48:47.472290 10952 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 14:48:47.472316 10950 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 14:48:47.473104 10944 server_base.cc:1034] running on GCE node
I20260620 14:48:47.473827 10944 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 14:48:47.474834 10944 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 14:48:47.476038 10944 hybrid_clock.cc:648] HybridClock initialized: now 1781966927475964 us; error 64 us; skew 500 ppm
I20260620 14:48:47.476373 10944 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 14:48:47 dist-test-slave-q9j6 krb5kdc[10703](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781966927, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 14:48:47.479776 10944 init.cc:377] Logged in from keytab as kudu/127.0.97.2@KRBTEST.COM (short username kudu)
I20260620 14:48:47.481241 10944 webserver.cc:460] Webserver started at http://127.0.97.2:43253/ using document root <none> and password file <none>
I20260620 14:48:47.481909 10944 fs_manager.cc:359] Metadata directory not provided
I20260620 14:48:47.482004 10944 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 14:48:47.482244 10944 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 14:48:47.485278 10944 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskUci3DU/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781966829091300-388-0/minicluster-data/ts-1/data/instance:
uuid: "3ac7ff796e034848b5e46a3bc03d1c62"
format_stamp: "Formatted at 2026-06-20 14:48:47 on dist-test-slave-q9j6"
server_key: "4bb9901d9c0b62c7e4d9c851707fc7a4"
server_key_iv: "7b746caa995c428cd0abd0a7d104a790"
server_key_version: "encryptionkey@0"
I20260620 14:48:47.486038 10944 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskUci3DU/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781966829091300-388-0/minicluster-data/ts-1/wal/instance:
uuid: "3ac7ff796e034848b5e46a3bc03d1c62"
format_stamp: "Formatted at 2026-06-20 14:48:47 on dist-test-slave-q9j6"
server_key: "4bb9901d9c0b62c7e4d9c851707fc7a4"
server_key_iv: "7b746caa995c428cd0abd0a7d104a790"
server_key_version: "encryptionkey@0"
I20260620 14:48:47.491314 10944 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.002s sys 0.004s
I20260620 14:48:47.494889 10959 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 14:48:47.496515 10944 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260620 14:48:47.496692 10944 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskUci3DU/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781966829091300-388-0/minicluster-data/ts-1/data,/tmp/dist-test-taskUci3DU/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781966829091300-388-0/minicluster-data/ts-1/wal
uuid: "3ac7ff796e034848b5e46a3bc03d1c62"
format_stamp: "Formatted at 2026-06-20 14:48:47 on dist-test-slave-q9j6"
server_key: "4bb9901d9c0b62c7e4d9c851707fc7a4"
server_key_iv: "7b746caa995c428cd0abd0a7d104a790"
server_key_version: "encryptionkey@0"
I20260620 14:48:47.496927 10944 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskUci3DU/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781966829091300-388-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskUci3DU/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781966829091300-388-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskUci3DU/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781966829091300-388-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 14:48:47.528638 10944 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 14:48:47.532914 10944 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 14:48:47.533228 10944 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 14:48:47.534228 10944 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 14:48:47.535653 10944 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 14:48:47.535738 10944 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 14:48:47.535905 10944 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 14:48:47.535985 10944 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 14:48:47.569314 10944 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.2:40449
I20260620 14:48:47.569361 11071 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.2:40449 every 8 connection(s)
I20260620 14:48:47.570668 10944 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskUci3DU/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781966829091300-388-0/minicluster-data/ts-1/data/info.pb
I20260620 14:48:47.580269 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskUci3DU/build/asan/bin/kudu as pid 10944
I20260620 14:48:47.580454 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskUci3DU/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781966829091300-388-0/minicluster-data/ts-1/wal/instance
I20260620 14:48:47.580945 388 external_mini_cluster.cc:1428] Setting key 6193ba37b62148edcef3e27b5a55ed8e
Jun 20 14:48:47 dist-test-slave-q9j6 krb5kdc[10703](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781966927, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.2@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260620 14:48:47.592326 11074 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:48:47.573260 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.2:58419 (local address 127.0.97.62:41487)
0620 14:48:47.573729 (+ 469us) server_negotiation.cc:206] Beginning negotiation
0620 14:48:47.573735 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 14:48:47.574770 (+ 1035us) server_negotiation.cc:407] Connection header received
0620 14:48:47.576631 (+ 1861us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:48:47.576635 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:48:47.576707 (+ 72us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:48:47.576810 (+ 103us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:48:47.578297 (+ 1487us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:48:47.579076 (+ 779us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:48:47.580906 (+ 1830us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:48:47.581571 (+ 665us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:48:47.581675 (+ 104us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:48:47.585596 (+ 3921us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 14:48:47.585605 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 14:48:47.585613 (+ 8us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 14:48:47.585662 (+ 49us) server_negotiation.cc:922] calling sasl_server_start()
0620 14:48:47.589080 (+ 3418us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:48:47.589842 (+ 762us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:48:47.589850 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:48:47.589856 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:48:47.590010 (+ 154us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:48:47.590415 (+ 405us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:48:47.590423 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:48:47.590429 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:48:47.590806 (+ 377us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 14:48:47.590962 (+ 156us) server_negotiation.cc:1035] Waiting for connection context
0620 14:48:47.591887 (+ 925us) server_negotiation.cc:299] Negotiation successful
0620 14:48:47.592053 (+ 166us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":347,"thread_start_us":224,"threads_started":1}
I20260620 14:48:47.593914 11075 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:48:47.573894 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:41487 (local address 127.0.97.2:58419)
0620 14:48:47.574594 (+ 700us) negotiation.cc:107] Waiting for socket to connect
0620 14:48:47.574636 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0620 14:48:47.576351 (+ 1715us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 14:48:47.577029 (+ 678us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 14:48:47.577038 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 14:48:47.577558 (+ 520us) client_negotiation.cc:189] Negotiated authn=SASL
0620 14:48:47.578129 (+ 571us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:48:47.578145 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:48:47.579246 (+ 1101us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:48:47.579252 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:48:47.580755 (+ 1503us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:48:47.580768 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:48:47.581804 (+ 1036us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:48:47.581814 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:48:47.582030 (+ 216us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:48:47.582773 (+ 743us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 14:48:47.582809 (+ 36us) client_negotiation.cc:656] calling sasl_client_start()
0620 14:48:47.585332 (+ 2523us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 14:48:47.589239 (+ 3907us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 14:48:47.589247 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 14:48:47.589261 (+ 14us) client_negotiation.cc:760] calling sasl_client_step()
0620 14:48:47.589701 (+ 440us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 14:48:47.590115 (+ 414us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 14:48:47.590118 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 14:48:47.590121 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0620 14:48:47.590299 (+ 178us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 14:48:47.590956 (+ 657us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 14:48:47.590968 (+ 12us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 14:48:47.591460 (+ 492us) client_negotiation.cc:769] Sending connection context
0620 14:48:47.591820 (+ 360us) client_negotiation.cc:240] Negotiation successful
0620 14:48:47.592421 (+ 601us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":456,"thread_start_us":290,"threads_started":1}
I20260620 14:48:47.595450 11072 heartbeater.cc:347] Connected to a master server at 127.0.97.62:41487
I20260620 14:48:47.595814 11072 heartbeater.cc:464] Registering TS with master...
I20260620 14:48:47.596585 11072 heartbeater.cc:511] Master 127.0.97.62:41487 requested a full tablet report, sending...
I20260620 14:48:47.598124 10748 ts_manager.cc:194] Registered new tserver with Master: 3ac7ff796e034848b5e46a3bc03d1c62 (127.0.97.2:40449)
I20260620 14:48:47.598945 10748 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.97.2@KRBTEST.COM'} at 127.0.97.2:58419
WARNING: no policy specified for kudu/127.0.97.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskUci3DU/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781966829091300-388-0/krb5kdc/kudu_127.0.97.3.keytab.
Entry for principal kudu/127.0.97.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskUci3DU/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781966829091300-388-0/krb5kdc/kudu_127.0.97.3.keytab.
I20260620 14:48:47.634366 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskUci3DU/build/asan/bin/kudu
/tmp/dist-test-taskUci3DU/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskUci3DU/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781966829091300-388-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskUci3DU/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781966829091300-388-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskUci3DU/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781966829091300-388-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskUci3DU/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781966829091300-388-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.3:0
--local_ip_for_outbound_sockets=127.0.97.3
--webserver_interface=127.0.97.3
--webserver_port=0
--tserver_master_addrs=127.0.97.62:41487
--keytab_file=/tmp/dist-test-taskUci3DU/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781966829091300-388-0/krb5kdc/kudu_127.0.97.3.keytab
--principal=kudu/127.0.97.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:34299
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskUci3DU/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781966829091300-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskUci3DU/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781966829091300-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskUci3DU/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781966829091300-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 14:48:47.966084 11078 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 14:48:47.966409 11078 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 14:48:47.966596 11078 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 14:48:47.974309 11078 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 14:48:47.974445 11078 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 14:48:47.974603 11078 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.3
I20260620 14:48:47.985023 11078 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:34299
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskUci3DU/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781966829091300-388-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskUci3DU/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781966829091300-388-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskUci3DU/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781966829091300-388-0/krb5kdc/kudu_127.0.97.3.keytab
--principal=kudu/127.0.97.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskUci3DU/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781966829091300-388-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.97.3
--webserver_port=0
--tserver_master_addrs=127.0.97.62:41487
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.3
--log_dir=/tmp/dist-test-taskUci3DU/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781966829091300-388-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 13:41:20 UTC on ccycloud.quasar-vdagoi.root.comops.site
build id 4457462
ASAN enabled
I20260620 14:48:47.986446 11078 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 14:48:47.987845 11078 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 14:48:48.001924 11083 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 14:48:48.001951 11084 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 14:48:48.001924 11086 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 14:48:48.003065 11078 server_base.cc:1034] running on GCE node
I20260620 14:48:48.003821 11078 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 14:48:48.004870 11078 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 14:48:48.006078 11078 hybrid_clock.cc:648] HybridClock initialized: now 1781966928006014 us; error 50 us; skew 500 ppm
I20260620 14:48:48.006439 11078 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 14:48:48 dist-test-slave-q9j6 krb5kdc[10703](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781966928, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 14:48:48.009801 11078 init.cc:377] Logged in from keytab as kudu/127.0.97.3@KRBTEST.COM (short username kudu)
I20260620 14:48:48.011323 11078 webserver.cc:460] Webserver started at http://127.0.97.3:45449/ using document root <none> and password file <none>
I20260620 14:48:48.012133 11078 fs_manager.cc:359] Metadata directory not provided
I20260620 14:48:48.012269 11078 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 14:48:48.012614 11078 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 14:48:48.015795 11078 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskUci3DU/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781966829091300-388-0/minicluster-data/ts-2/data/instance:
uuid: "caf7bbda41c44c57b48367e7eda0d776"
format_stamp: "Formatted at 2026-06-20 14:48:48 on dist-test-slave-q9j6"
server_key: "be3a353458f97387ef10e8b92909dc39"
server_key_iv: "e738603844d53668f765a2a56dac4d1c"
server_key_version: "encryptionkey@0"
I20260620 14:48:48.016659 11078 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskUci3DU/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781966829091300-388-0/minicluster-data/ts-2/wal/instance:
uuid: "caf7bbda41c44c57b48367e7eda0d776"
format_stamp: "Formatted at 2026-06-20 14:48:48 on dist-test-slave-q9j6"
server_key: "be3a353458f97387ef10e8b92909dc39"
server_key_iv: "e738603844d53668f765a2a56dac4d1c"
server_key_version: "encryptionkey@0"
I20260620 14:48:48.022249 11078 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.002s sys 0.004s
I20260620 14:48:48.026026 11093 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 14:48:48.027845 11078 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260620 14:48:48.028083 11078 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskUci3DU/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781966829091300-388-0/minicluster-data/ts-2/data,/tmp/dist-test-taskUci3DU/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781966829091300-388-0/minicluster-data/ts-2/wal
uuid: "caf7bbda41c44c57b48367e7eda0d776"
format_stamp: "Formatted at 2026-06-20 14:48:48 on dist-test-slave-q9j6"
server_key: "be3a353458f97387ef10e8b92909dc39"
server_key_iv: "e738603844d53668f765a2a56dac4d1c"
server_key_version: "encryptionkey@0"
I20260620 14:48:48.028307 11078 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskUci3DU/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781966829091300-388-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskUci3DU/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781966829091300-388-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskUci3DU/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781966829091300-388-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 14:48:48.049849 11078 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 14:48:48.053948 11078 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 14:48:48.054302 11078 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 14:48:48.055490 11078 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 14:48:48.057180 11078 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 14:48:48.057256 11078 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 14:48:48.057441 11078 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 14:48:48.057494 11078 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 14:48:48.074779 10938 heartbeater.cc:503] Master 127.0.97.62:41487 was elected leader, sending a full tablet report...
I20260620 14:48:48.091411 11078 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.3:37789
I20260620 14:48:48.091460 11205 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.3:37789 every 8 connection(s)
I20260620 14:48:48.092841 11078 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskUci3DU/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781966829091300-388-0/minicluster-data/ts-2/data/info.pb
I20260620 14:48:48.097453 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskUci3DU/build/asan/bin/kudu as pid 11078
I20260620 14:48:48.097651 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskUci3DU/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781966829091300-388-0/minicluster-data/ts-2/wal/instance
I20260620 14:48:48.098073 388 external_mini_cluster.cc:1428] Setting key 94101f1e72d359adc53ac2930323f613
Jun 20 14:48:48 dist-test-slave-q9j6 krb5kdc[10703](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781966928, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.3@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260620 14:48:48.114406 11208 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:48:48.095644 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.3:43749 (local address 127.0.97.62:41487)
0620 14:48:48.096131 (+ 487us) server_negotiation.cc:206] Beginning negotiation
0620 14:48:48.096136 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 14:48:48.097065 (+ 929us) server_negotiation.cc:407] Connection header received
0620 14:48:48.099151 (+ 2086us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:48:48.099157 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:48:48.099293 (+ 136us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:48:48.099402 (+ 109us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:48:48.101677 (+ 2275us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:48:48.102435 (+ 758us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:48:48.104088 (+ 1653us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:48:48.104479 (+ 391us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:48:48.104577 (+ 98us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:48:48.108048 (+ 3471us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 14:48:48.108053 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 14:48:48.108056 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 14:48:48.108090 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0620 14:48:48.110836 (+ 2746us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:48:48.111660 (+ 824us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:48:48.111665 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:48:48.111668 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:48:48.111825 (+ 157us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:48:48.112291 (+ 466us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:48:48.112296 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:48:48.112301 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:48:48.112647 (+ 346us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 14:48:48.112809 (+ 162us) server_negotiation.cc:1035] Waiting for connection context
0620 14:48:48.113764 (+ 955us) server_negotiation.cc:299] Negotiation successful
0620 14:48:48.114036 (+ 272us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":379,"thread_start_us":246,"threads_started":1}
I20260620 14:48:48.115443 11209 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:48:48.096176 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:41487 (local address 127.0.97.3:43749)
0620 14:48:48.096886 (+ 710us) negotiation.cc:107] Waiting for socket to connect
0620 14:48:48.096929 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0620 14:48:48.098802 (+ 1873us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 14:48:48.099763 (+ 961us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 14:48:48.099773 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 14:48:48.100641 (+ 868us) client_negotiation.cc:189] Negotiated authn=SASL
0620 14:48:48.101512 (+ 871us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:48:48.101533 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:48:48.102558 (+ 1025us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:48:48.102562 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:48:48.103945 (+ 1383us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:48:48.103958 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:48:48.104673 (+ 715us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:48:48.104678 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:48:48.104876 (+ 198us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:48:48.105541 (+ 665us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 14:48:48.105567 (+ 26us) client_negotiation.cc:656] calling sasl_client_start()
0620 14:48:48.107811 (+ 2244us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 14:48:48.111009 (+ 3198us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 14:48:48.111017 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 14:48:48.111032 (+ 15us) client_negotiation.cc:760] calling sasl_client_step()
0620 14:48:48.111497 (+ 465us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 14:48:48.111949 (+ 452us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 14:48:48.111952 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 14:48:48.111955 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0620 14:48:48.112163 (+ 208us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 14:48:48.112822 (+ 659us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 14:48:48.112831 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 14:48:48.113288 (+ 457us) client_negotiation.cc:769] Sending connection context
0620 14:48:48.113633 (+ 345us) client_negotiation.cc:240] Negotiation successful
0620 14:48:48.114240 (+ 607us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":478,"thread_start_us":278,"threads_started":1}
I20260620 14:48:48.116923 11206 heartbeater.cc:347] Connected to a master server at 127.0.97.62:41487
I20260620 14:48:48.117261 11206 heartbeater.cc:464] Registering TS with master...
I20260620 14:48:48.118005 11206 heartbeater.cc:511] Master 127.0.97.62:41487 requested a full tablet report, sending...
I20260620 14:48:48.119554 10748 ts_manager.cc:194] Registered new tserver with Master: caf7bbda41c44c57b48367e7eda0d776 (127.0.97.3:37789)
I20260620 14:48:48.120463 10748 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.97.3@KRBTEST.COM'} at 127.0.97.3:43749
I20260620 14:48:48.126590 388 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260620 14:48:48.144237 11208 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:48:48.131441 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36398 (local address 127.0.97.62:41487)
0620 14:48:48.131722 (+ 281us) server_negotiation.cc:206] Beginning negotiation
0620 14:48:48.131731 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0620 14:48:48.132043 (+ 312us) server_negotiation.cc:407] Connection header received
0620 14:48:48.132286 (+ 243us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:48:48.132290 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:48:48.132365 (+ 75us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:48:48.132476 (+ 111us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:48:48.133864 (+ 1388us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:48:48.134598 (+ 734us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:48:48.135910 (+ 1312us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:48:48.136337 (+ 427us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:48:48.136481 (+ 144us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:48:48.139185 (+ 2704us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 14:48:48.139191 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 14:48:48.139194 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 14:48:48.139228 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0620 14:48:48.141568 (+ 2340us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:48:48.142206 (+ 638us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:48:48.142212 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:48:48.142218 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:48:48.142338 (+ 120us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:48:48.142723 (+ 385us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:48:48.142728 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:48:48.142731 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:48:48.143071 (+ 340us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 14:48:48.143330 (+ 259us) server_negotiation.cc:1035] Waiting for connection context
0620 14:48:48.143698 (+ 368us) server_negotiation.cc:299] Negotiation successful
0620 14:48:48.143927 (+ 229us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":95}
I20260620 14:48:48.148248 10748 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:36398:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260620 14:48:48.155021 10748 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260620 14:48:48.180716 11221 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:48:48.166877 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:45285 (local address 127.0.0.1:34954)
0620 14:48:48.168390 (+ 1513us) negotiation.cc:107] Waiting for socket to connect
0620 14:48:48.168475 (+ 85us) client_negotiation.cc:174] Beginning negotiation
0620 14:48:48.168664 (+ 189us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 14:48:48.169844 (+ 1180us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 14:48:48.169855 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 14:48:48.169897 (+ 42us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 14:48:48.170411 (+ 514us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:48:48.170426 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:48:48.173065 (+ 2639us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:48:48.173073 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:48:48.176231 (+ 3158us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:48:48.176253 (+ 22us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:48:48.179330 (+ 3077us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:48:48.179340 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:48:48.179655 (+ 315us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:48:48.179729 (+ 74us) client_negotiation.cc:769] Sending connection context
0620 14:48:48.179879 (+ 150us) client_negotiation.cc:240] Negotiation successful
0620 14:48:48.180181 (+ 302us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1255,"thread_start_us":1390,"threads_started":1}
I20260620 14:48:48.181897 11220 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:48:48.167042 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34954 (local address 127.0.97.1:45285)
0620 14:48:48.167757 (+ 715us) server_negotiation.cc:206] Beginning negotiation
0620 14:48:48.167765 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0620 14:48:48.168583 (+ 818us) server_negotiation.cc:407] Connection header received
0620 14:48:48.168916 (+ 333us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:48:48.168930 (+ 14us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:48:48.169269 (+ 339us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:48:48.169463 (+ 194us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 14:48:48.170669 (+ 1206us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:48:48.172836 (+ 2167us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:48:48.176487 (+ 3651us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:48:48.178606 (+ 2119us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:48:48.178738 (+ 132us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:48:48.179072 (+ 334us) server_negotiation.cc:1035] Waiting for connection context
0620 14:48:48.181209 (+ 2137us) server_negotiation.cc:299] Negotiation successful
0620 14:48:48.181465 (+ 256us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":504,"thread_start_us":340,"threads_started":1}
I20260620 14:48:48.187579 10873 tablet_service.cc:1467] Processing CreateTablet for tablet 1ef26d2ae62044a7ab7731912bd2617d (DEFAULT_TABLE table=test-table [id=2c3f74c91d9a49e19e9dc23dec76fc06]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 14:48:48.189116 10873 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1ef26d2ae62044a7ab7731912bd2617d. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 14:48:48.190456 11219 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:48:48.167625 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.3:37789 (local address 127.0.0.1:45338)
0620 14:48:48.168390 (+ 765us) negotiation.cc:107] Waiting for socket to connect
0620 14:48:48.168475 (+ 85us) client_negotiation.cc:174] Beginning negotiation
0620 14:48:48.168664 (+ 189us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 14:48:48.183690 (+ 15026us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 14:48:48.183697 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 14:48:48.183728 (+ 31us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 14:48:48.184039 (+ 311us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:48:48.184049 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:48:48.186268 (+ 2219us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:48:48.186273 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:48:48.188043 (+ 1770us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:48:48.188054 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:48:48.189616 (+ 1562us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:48:48.189623 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:48:48.189794 (+ 171us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:48:48.189827 (+ 33us) client_negotiation.cc:769] Sending connection context
0620 14:48:48.189924 (+ 97us) client_negotiation.cc:240] Negotiation successful
0620 14:48:48.190050 (+ 126us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":609,"mutex_wait_us":54,"thread_start_us":322,"threads_started":1}
I20260620 14:48:48.190496 11222 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:48:48.168040 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45338 (local address 127.0.97.3:37789)
0620 14:48:48.182919 (+ 14879us) server_negotiation.cc:206] Beginning negotiation
0620 14:48:48.182928 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0620 14:48:48.182988 (+ 60us) server_negotiation.cc:407] Connection header received
0620 14:48:48.183106 (+ 118us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:48:48.183116 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:48:48.183427 (+ 311us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:48:48.184237 (+ 810us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 14:48:48.184537 (+ 300us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:48:48.186124 (+ 1587us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:48:48.188202 (+ 2078us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:48:48.189418 (+ 1216us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:48:48.189546 (+ 128us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:48:48.189786 (+ 240us) server_negotiation.cc:1035] Waiting for connection context
0620 14:48:48.189981 (+ 195us) server_negotiation.cc:299] Negotiation successful
0620 14:48:48.190158 (+ 177us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":14705,"thread_start_us":14549,"threads_started":1}
I20260620 14:48:48.191242 11221 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:48:48.168806 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.2:40449 (local address 127.0.0.1:46762)
0620 14:48:48.183050 (+ 14244us) negotiation.cc:107] Waiting for socket to connect
0620 14:48:48.183070 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0620 14:48:48.183169 (+ 99us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 14:48:48.184566 (+ 1397us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 14:48:48.184570 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 14:48:48.184611 (+ 41us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 14:48:48.184810 (+ 199us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:48:48.184822 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:48:48.186842 (+ 2020us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:48:48.186847 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:48:48.188732 (+ 1885us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:48:48.188748 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:48:48.190502 (+ 1754us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:48:48.190509 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:48:48.190702 (+ 193us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:48:48.190744 (+ 42us) client_negotiation.cc:769] Sending connection context
0620 14:48:48.190811 (+ 67us) client_negotiation.cc:240] Negotiation successful
0620 14:48:48.190942 (+ 131us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":14125,"mutex_wait_us":3,"thread_start_us":14586,"threads_started":1}
I20260620 14:48:48.192266 11224 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:48:48.169058 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46762 (local address 127.0.97.2:40449)
0620 14:48:48.183953 (+ 14895us) server_negotiation.cc:206] Beginning negotiation
0620 14:48:48.183963 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0620 14:48:48.183992 (+ 29us) server_negotiation.cc:407] Connection header received
0620 14:48:48.184097 (+ 105us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:48:48.184107 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:48:48.184353 (+ 246us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:48:48.184988 (+ 635us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 14:48:48.185184 (+ 196us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:48:48.186687 (+ 1503us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:48:48.188907 (+ 2220us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:48:48.190316 (+ 1409us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:48:48.191490 (+ 1174us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:48:48.191713 (+ 223us) server_negotiation.cc:1035] Waiting for connection context
0620 14:48:48.191788 (+ 75us) server_negotiation.cc:299] Negotiation successful
0620 14:48:48.191943 (+ 155us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":14730,"thread_start_us":14627,"threads_started":1}
I20260620 14:48:48.194096 11141 tablet_service.cc:1467] Processing CreateTablet for tablet 1ef26d2ae62044a7ab7731912bd2617d (DEFAULT_TABLE table=test-table [id=2c3f74c91d9a49e19e9dc23dec76fc06]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 14:48:48.195685 11141 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1ef26d2ae62044a7ab7731912bd2617d. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 14:48:48.196062 11007 tablet_service.cc:1467] Processing CreateTablet for tablet 1ef26d2ae62044a7ab7731912bd2617d (DEFAULT_TABLE table=test-table [id=2c3f74c91d9a49e19e9dc23dec76fc06]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 14:48:48.197366 11007 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1ef26d2ae62044a7ab7731912bd2617d. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 14:48:48.202220 11225 tablet_bootstrap.cc:492] T 1ef26d2ae62044a7ab7731912bd2617d P 380b5d5555c244899f8a06998284b5a0: Bootstrap starting.
I20260620 14:48:48.205397 11225 tablet_bootstrap.cc:654] T 1ef26d2ae62044a7ab7731912bd2617d P 380b5d5555c244899f8a06998284b5a0: Neither blocks nor log segments found. Creating new log.
I20260620 14:48:48.206430 11225 log.cc:824] T 1ef26d2ae62044a7ab7731912bd2617d P 380b5d5555c244899f8a06998284b5a0: Log is configured to *not* fsync() on all Append() calls
I20260620 14:48:48.209365 11225 tablet_bootstrap.cc:492] T 1ef26d2ae62044a7ab7731912bd2617d P 380b5d5555c244899f8a06998284b5a0: No bootstrap required, opened a new log
I20260620 14:48:48.209683 11225 ts_tablet_manager.cc:1397] T 1ef26d2ae62044a7ab7731912bd2617d P 380b5d5555c244899f8a06998284b5a0: Time spent bootstrapping tablet: real 0.008s user 0.000s sys 0.005s
I20260620 14:48:48.210335 11227 tablet_bootstrap.cc:492] T 1ef26d2ae62044a7ab7731912bd2617d P caf7bbda41c44c57b48367e7eda0d776: Bootstrap starting.
I20260620 14:48:48.211925 11228 tablet_bootstrap.cc:492] T 1ef26d2ae62044a7ab7731912bd2617d P 3ac7ff796e034848b5e46a3bc03d1c62: Bootstrap starting.
I20260620 14:48:48.213986 11227 tablet_bootstrap.cc:654] T 1ef26d2ae62044a7ab7731912bd2617d P caf7bbda41c44c57b48367e7eda0d776: Neither blocks nor log segments found. Creating new log.
I20260620 14:48:48.214670 11228 tablet_bootstrap.cc:654] T 1ef26d2ae62044a7ab7731912bd2617d P 3ac7ff796e034848b5e46a3bc03d1c62: Neither blocks nor log segments found. Creating new log.
I20260620 14:48:48.215241 11227 log.cc:824] T 1ef26d2ae62044a7ab7731912bd2617d P caf7bbda41c44c57b48367e7eda0d776: Log is configured to *not* fsync() on all Append() calls
I20260620 14:48:48.215721 11228 log.cc:824] T 1ef26d2ae62044a7ab7731912bd2617d P 3ac7ff796e034848b5e46a3bc03d1c62: Log is configured to *not* fsync() on all Append() calls
I20260620 14:48:48.218382 11227 tablet_bootstrap.cc:492] T 1ef26d2ae62044a7ab7731912bd2617d P caf7bbda41c44c57b48367e7eda0d776: No bootstrap required, opened a new log
I20260620 14:48:48.218664 11228 tablet_bootstrap.cc:492] T 1ef26d2ae62044a7ab7731912bd2617d P 3ac7ff796e034848b5e46a3bc03d1c62: No bootstrap required, opened a new log
I20260620 14:48:48.218676 11227 ts_tablet_manager.cc:1397] T 1ef26d2ae62044a7ab7731912bd2617d P caf7bbda41c44c57b48367e7eda0d776: Time spent bootstrapping tablet: real 0.009s user 0.008s sys 0.000s
I20260620 14:48:48.218904 11228 ts_tablet_manager.cc:1397] T 1ef26d2ae62044a7ab7731912bd2617d P 3ac7ff796e034848b5e46a3bc03d1c62: Time spent bootstrapping tablet: real 0.007s user 0.005s sys 0.000s
I20260620 14:48:48.220006 11225 raft_consensus.cc:348] T 1ef26d2ae62044a7ab7731912bd2617d P 380b5d5555c244899f8a06998284b5a0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "380b5d5555c244899f8a06998284b5a0" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 45285 } } peers { permanent_uuid: "3ac7ff796e034848b5e46a3bc03d1c62" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 40449 } } peers { permanent_uuid: "caf7bbda41c44c57b48367e7eda0d776" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 37789 } }
I20260620 14:48:48.220382 11225 raft_consensus.cc:374] T 1ef26d2ae62044a7ab7731912bd2617d P 380b5d5555c244899f8a06998284b5a0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 14:48:48.220541 11225 raft_consensus.cc:729] T 1ef26d2ae62044a7ab7731912bd2617d P 380b5d5555c244899f8a06998284b5a0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 380b5d5555c244899f8a06998284b5a0, State: Initialized, Role: FOLLOWER
I20260620 14:48:48.221149 11225 consensus_queue.cc:260] T 1ef26d2ae62044a7ab7731912bd2617d P 380b5d5555c244899f8a06998284b5a0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "380b5d5555c244899f8a06998284b5a0" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 45285 } } peers { permanent_uuid: "3ac7ff796e034848b5e46a3bc03d1c62" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 40449 } } peers { permanent_uuid: "caf7bbda41c44c57b48367e7eda0d776" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 37789 } }
I20260620 14:48:48.222918 11225 ts_tablet_manager.cc:1428] T 1ef26d2ae62044a7ab7731912bd2617d P 380b5d5555c244899f8a06998284b5a0: Time spent starting tablet: real 0.013s user 0.011s sys 0.000s
I20260620 14:48:48.229540 11228 raft_consensus.cc:348] T 1ef26d2ae62044a7ab7731912bd2617d P 3ac7ff796e034848b5e46a3bc03d1c62 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "380b5d5555c244899f8a06998284b5a0" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 45285 } } peers { permanent_uuid: "3ac7ff796e034848b5e46a3bc03d1c62" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 40449 } } peers { permanent_uuid: "caf7bbda41c44c57b48367e7eda0d776" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 37789 } }
I20260620 14:48:48.229540 11227 raft_consensus.cc:348] T 1ef26d2ae62044a7ab7731912bd2617d P caf7bbda41c44c57b48367e7eda0d776 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "380b5d5555c244899f8a06998284b5a0" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 45285 } } peers { permanent_uuid: "3ac7ff796e034848b5e46a3bc03d1c62" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 40449 } } peers { permanent_uuid: "caf7bbda41c44c57b48367e7eda0d776" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 37789 } }
I20260620 14:48:48.230014 11228 raft_consensus.cc:374] T 1ef26d2ae62044a7ab7731912bd2617d P 3ac7ff796e034848b5e46a3bc03d1c62 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 14:48:48.230014 11227 raft_consensus.cc:374] T 1ef26d2ae62044a7ab7731912bd2617d P caf7bbda41c44c57b48367e7eda0d776 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 14:48:48.230125 11228 raft_consensus.cc:729] T 1ef26d2ae62044a7ab7731912bd2617d P 3ac7ff796e034848b5e46a3bc03d1c62 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3ac7ff796e034848b5e46a3bc03d1c62, State: Initialized, Role: FOLLOWER
I20260620 14:48:48.230121 11227 raft_consensus.cc:729] T 1ef26d2ae62044a7ab7731912bd2617d P caf7bbda41c44c57b48367e7eda0d776 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: caf7bbda41c44c57b48367e7eda0d776, State: Initialized, Role: FOLLOWER
I20260620 14:48:48.230774 11227 consensus_queue.cc:260] T 1ef26d2ae62044a7ab7731912bd2617d P caf7bbda41c44c57b48367e7eda0d776 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "380b5d5555c244899f8a06998284b5a0" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 45285 } } peers { permanent_uuid: "3ac7ff796e034848b5e46a3bc03d1c62" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 40449 } } peers { permanent_uuid: "caf7bbda41c44c57b48367e7eda0d776" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 37789 } }
I20260620 14:48:48.230774 11228 consensus_queue.cc:260] T 1ef26d2ae62044a7ab7731912bd2617d P 3ac7ff796e034848b5e46a3bc03d1c62 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "380b5d5555c244899f8a06998284b5a0" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 45285 } } peers { permanent_uuid: "3ac7ff796e034848b5e46a3bc03d1c62" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 40449 } } peers { permanent_uuid: "caf7bbda41c44c57b48367e7eda0d776" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 37789 } }
I20260620 14:48:48.232841 11227 ts_tablet_manager.cc:1428] T 1ef26d2ae62044a7ab7731912bd2617d P caf7bbda41c44c57b48367e7eda0d776: Time spent starting tablet: real 0.014s user 0.014s sys 0.000s
I20260620 14:48:48.232824 11072 heartbeater.cc:503] Master 127.0.97.62:41487 was elected leader, sending a full tablet report...
I20260620 14:48:48.232846 11228 ts_tablet_manager.cc:1428] T 1ef26d2ae62044a7ab7731912bd2617d P 3ac7ff796e034848b5e46a3bc03d1c62: Time spent starting tablet: real 0.014s user 0.015s sys 0.000s
I20260620 14:48:48.233273 11206 heartbeater.cc:503] Master 127.0.97.62:41487 was elected leader, sending a full tablet report...
I20260620 14:48:48.246155 11232 raft_consensus.cc:482] T 1ef26d2ae62044a7ab7731912bd2617d P 3ac7ff796e034848b5e46a3bc03d1c62 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260620 14:48:48.246483 11232 raft_consensus.cc:504] T 1ef26d2ae62044a7ab7731912bd2617d P 3ac7ff796e034848b5e46a3bc03d1c62 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "380b5d5555c244899f8a06998284b5a0" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 45285 } } peers { permanent_uuid: "3ac7ff796e034848b5e46a3bc03d1c62" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 40449 } } peers { permanent_uuid: "caf7bbda41c44c57b48367e7eda0d776" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 37789 } }
I20260620 14:48:48.248363 11232 leader_election.cc:288] T 1ef26d2ae62044a7ab7731912bd2617d P 3ac7ff796e034848b5e46a3bc03d1c62 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 380b5d5555c244899f8a06998284b5a0 (127.0.97.1:45285), caf7bbda41c44c57b48367e7eda0d776 (127.0.97.3:37789)
I20260620 14:48:48.255316 11222 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:48:48.248809 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.2:57117 (local address 127.0.97.3:37789)
0620 14:48:48.249040 (+ 231us) server_negotiation.cc:206] Beginning negotiation
0620 14:48:48.249048 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0620 14:48:48.249472 (+ 424us) server_negotiation.cc:407] Connection header received
0620 14:48:48.249768 (+ 296us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:48:48.249776 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:48:48.249857 (+ 81us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:48:48.249947 (+ 90us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 14:48:48.250566 (+ 619us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:48:48.251379 (+ 813us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:48:48.253298 (+ 1919us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:48:48.254242 (+ 944us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:48:48.254371 (+ 129us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:48:48.254489 (+ 118us) server_negotiation.cc:1035] Waiting for connection context
0620 14:48:48.254980 (+ 491us) server_negotiation.cc:299] Negotiation successful
0620 14:48:48.255115 (+ 135us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":79}
I20260620 14:48:48.255330 11220 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:48:48.249287 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.2:49537 (local address 127.0.97.1:45285)
0620 14:48:48.249490 (+ 203us) server_negotiation.cc:206] Beginning negotiation
0620 14:48:48.249496 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 14:48:48.249825 (+ 329us) server_negotiation.cc:407] Connection header received
0620 14:48:48.249987 (+ 162us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:48:48.249992 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:48:48.250071 (+ 79us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:48:48.250187 (+ 116us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 14:48:48.250739 (+ 552us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:48:48.251511 (+ 772us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:48:48.253405 (+ 1894us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:48:48.254256 (+ 851us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:48:48.254412 (+ 156us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:48:48.254520 (+ 108us) server_negotiation.cc:1035] Waiting for connection context
0620 14:48:48.254945 (+ 425us) server_negotiation.cc:299] Negotiation successful
0620 14:48:48.255082 (+ 137us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":65}
I20260620 14:48:48.255425 11235 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:48:48.248697 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:45285 (local address 127.0.97.2:49537)
0620 14:48:48.249709 (+ 1012us) negotiation.cc:107] Waiting for socket to connect
0620 14:48:48.249731 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0620 14:48:48.249849 (+ 118us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 14:48:48.250211 (+ 362us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 14:48:48.250214 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 14:48:48.250233 (+ 19us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 14:48:48.250582 (+ 349us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:48:48.250592 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:48:48.251699 (+ 1107us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:48:48.251705 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:48:48.253248 (+ 1543us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:48:48.253263 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:48:48.254520 (+ 1257us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:48:48.254527 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:48:48.254682 (+ 155us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:48:48.254807 (+ 125us) client_negotiation.cc:769] Sending connection context
0620 14:48:48.254917 (+ 110us) client_negotiation.cc:240] Negotiation successful
0620 14:48:48.255050 (+ 133us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":891,"mutex_wait_us":80,"thread_start_us":693,"threads_started":1}
I20260620 14:48:48.256143 11234 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:48:48.248590 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.3:37789 (local address 127.0.97.2:57117)
0620 14:48:48.249368 (+ 778us) negotiation.cc:107] Waiting for socket to connect
0620 14:48:48.249388 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0620 14:48:48.249602 (+ 214us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 14:48:48.249973 (+ 371us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 14:48:48.249978 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 14:48:48.250004 (+ 26us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 14:48:48.250421 (+ 417us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:48:48.250434 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:48:48.251511 (+ 1077us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:48:48.251518 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:48:48.253144 (+ 1626us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:48:48.253157 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:48:48.254477 (+ 1320us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:48:48.254483 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:48:48.254757 (+ 274us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:48:48.254814 (+ 57us) client_negotiation.cc:769] Sending connection context
0620 14:48:48.255524 (+ 710us) client_negotiation.cc:240] Negotiation successful
0620 14:48:48.255740 (+ 216us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":603,"mutex_wait_us":48,"thread_start_us":412,"threads_started":1}
I20260620 14:48:48.256820 10893 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1ef26d2ae62044a7ab7731912bd2617d" candidate_uuid: "3ac7ff796e034848b5e46a3bc03d1c62" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "380b5d5555c244899f8a06998284b5a0" is_pre_election: true
I20260620 14:48:48.257218 11161 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1ef26d2ae62044a7ab7731912bd2617d" candidate_uuid: "3ac7ff796e034848b5e46a3bc03d1c62" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "caf7bbda41c44c57b48367e7eda0d776" is_pre_election: true
I20260620 14:48:48.257483 10893 raft_consensus.cc:2454] T 1ef26d2ae62044a7ab7731912bd2617d P 380b5d5555c244899f8a06998284b5a0 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3ac7ff796e034848b5e46a3bc03d1c62 in term 0.
I20260620 14:48:48.257782 11161 raft_consensus.cc:2454] T 1ef26d2ae62044a7ab7731912bd2617d P caf7bbda41c44c57b48367e7eda0d776 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3ac7ff796e034848b5e46a3bc03d1c62 in term 0.
I20260620 14:48:48.258250 10963 leader_election.cc:302] T 1ef26d2ae62044a7ab7731912bd2617d P 3ac7ff796e034848b5e46a3bc03d1c62 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 380b5d5555c244899f8a06998284b5a0, 3ac7ff796e034848b5e46a3bc03d1c62; no voters:
I20260620 14:48:48.258647 11232 raft_consensus.cc:2781] T 1ef26d2ae62044a7ab7731912bd2617d P 3ac7ff796e034848b5e46a3bc03d1c62 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260620 14:48:48.258795 11232 raft_consensus.cc:482] T 1ef26d2ae62044a7ab7731912bd2617d P 3ac7ff796e034848b5e46a3bc03d1c62 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260620 14:48:48.258893 11232 raft_consensus.cc:3037] T 1ef26d2ae62044a7ab7731912bd2617d P 3ac7ff796e034848b5e46a3bc03d1c62 [term 0 FOLLOWER]: Advancing to term 1
I20260620 14:48:48.261147 11232 raft_consensus.cc:504] T 1ef26d2ae62044a7ab7731912bd2617d P 3ac7ff796e034848b5e46a3bc03d1c62 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "380b5d5555c244899f8a06998284b5a0" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 45285 } } peers { permanent_uuid: "3ac7ff796e034848b5e46a3bc03d1c62" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 40449 } } peers { permanent_uuid: "caf7bbda41c44c57b48367e7eda0d776" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 37789 } }
I20260620 14:48:48.261803 11232 leader_election.cc:288] T 1ef26d2ae62044a7ab7731912bd2617d P 3ac7ff796e034848b5e46a3bc03d1c62 [CANDIDATE]: Term 1 election: Requested vote from peers 380b5d5555c244899f8a06998284b5a0 (127.0.97.1:45285), caf7bbda41c44c57b48367e7eda0d776 (127.0.97.3:37789)
I20260620 14:48:48.262197 10893 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1ef26d2ae62044a7ab7731912bd2617d" candidate_uuid: "3ac7ff796e034848b5e46a3bc03d1c62" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "380b5d5555c244899f8a06998284b5a0"
I20260620 14:48:48.262393 11161 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1ef26d2ae62044a7ab7731912bd2617d" candidate_uuid: "3ac7ff796e034848b5e46a3bc03d1c62" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "caf7bbda41c44c57b48367e7eda0d776"
I20260620 14:48:48.262432 10893 raft_consensus.cc:3037] T 1ef26d2ae62044a7ab7731912bd2617d P 380b5d5555c244899f8a06998284b5a0 [term 0 FOLLOWER]: Advancing to term 1
I20260620 14:48:48.262611 11161 raft_consensus.cc:3037] T 1ef26d2ae62044a7ab7731912bd2617d P caf7bbda41c44c57b48367e7eda0d776 [term 0 FOLLOWER]: Advancing to term 1
I20260620 14:48:48.264760 10893 raft_consensus.cc:2454] T 1ef26d2ae62044a7ab7731912bd2617d P 380b5d5555c244899f8a06998284b5a0 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3ac7ff796e034848b5e46a3bc03d1c62 in term 1.
I20260620 14:48:48.264916 11161 raft_consensus.cc:2454] T 1ef26d2ae62044a7ab7731912bd2617d P caf7bbda41c44c57b48367e7eda0d776 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3ac7ff796e034848b5e46a3bc03d1c62 in term 1.
I20260620 14:48:48.265209 10963 leader_election.cc:302] T 1ef26d2ae62044a7ab7731912bd2617d P 3ac7ff796e034848b5e46a3bc03d1c62 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 380b5d5555c244899f8a06998284b5a0, 3ac7ff796e034848b5e46a3bc03d1c62; no voters:
I20260620 14:48:48.265465 11232 raft_consensus.cc:2781] T 1ef26d2ae62044a7ab7731912bd2617d P 3ac7ff796e034848b5e46a3bc03d1c62 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 14:48:48.265874 11232 raft_consensus.cc:686] T 1ef26d2ae62044a7ab7731912bd2617d P 3ac7ff796e034848b5e46a3bc03d1c62 [term 1 LEADER]: Becoming Leader. State: Replica: 3ac7ff796e034848b5e46a3bc03d1c62, State: Running, Role: LEADER
I20260620 14:48:48.266323 11232 consensus_queue.cc:237] T 1ef26d2ae62044a7ab7731912bd2617d P 3ac7ff796e034848b5e46a3bc03d1c62 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "380b5d5555c244899f8a06998284b5a0" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 45285 } } peers { permanent_uuid: "3ac7ff796e034848b5e46a3bc03d1c62" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 40449 } } peers { permanent_uuid: "caf7bbda41c44c57b48367e7eda0d776" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 37789 } }
I20260620 14:48:48.271225 10747 catalog_manager.cc:5495] T 1ef26d2ae62044a7ab7731912bd2617d P 3ac7ff796e034848b5e46a3bc03d1c62 reported cstate change: term changed from 0 to 1, leader changed from <none> to 3ac7ff796e034848b5e46a3bc03d1c62 (127.0.97.2). New cstate: current_term: 1 leader_uuid: "3ac7ff796e034848b5e46a3bc03d1c62" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "380b5d5555c244899f8a06998284b5a0" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 45285 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "3ac7ff796e034848b5e46a3bc03d1c62" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 40449 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "caf7bbda41c44c57b48367e7eda0d776" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 37789 } health_report { overall_health: UNKNOWN } } }
W20260620 14:48:48.289570 10939 tablet.cc:2369] T 1ef26d2ae62044a7ab7731912bd2617d P 380b5d5555c244899f8a06998284b5a0: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260620 14:48:48.304576 11224 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:48:48.299154 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46768 (local address 127.0.97.2:40449)
0620 14:48:48.299362 (+ 208us) server_negotiation.cc:206] Beginning negotiation
0620 14:48:48.299369 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0620 14:48:48.299453 (+ 84us) server_negotiation.cc:407] Connection header received
0620 14:48:48.299719 (+ 266us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:48:48.299723 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:48:48.299799 (+ 76us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:48:48.299954 (+ 155us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 14:48:48.300322 (+ 368us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:48:48.301101 (+ 779us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:48:48.302396 (+ 1295us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:48:48.302792 (+ 396us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:48:48.302925 (+ 133us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:48:48.303426 (+ 501us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 14:48:48.304003 (+ 577us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0620 14:48:48.304170 (+ 167us) server_negotiation.cc:1035] Waiting for connection context
0620 14:48:48.304221 (+ 51us) server_negotiation.cc:299] Negotiation successful
0620 14:48:48.304301 (+ 80us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":75}
I20260620 14:48:48.312858 11161 raft_consensus.cc:1261] T 1ef26d2ae62044a7ab7731912bd2617d P caf7bbda41c44c57b48367e7eda0d776 [term 1 FOLLOWER]: Refusing update from remote peer 3ac7ff796e034848b5e46a3bc03d1c62: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 14:48:48.312857 10893 raft_consensus.cc:1261] T 1ef26d2ae62044a7ab7731912bd2617d P 380b5d5555c244899f8a06998284b5a0 [term 1 FOLLOWER]: Refusing update from remote peer 3ac7ff796e034848b5e46a3bc03d1c62: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 14:48:48.313844 11236 consensus_queue.cc:1035] T 1ef26d2ae62044a7ab7731912bd2617d P 3ac7ff796e034848b5e46a3bc03d1c62 [LEADER]: Connected to new peer: Peer: permanent_uuid: "380b5d5555c244899f8a06998284b5a0" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 45285 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 14:48:48.314160 11232 consensus_queue.cc:1035] T 1ef26d2ae62044a7ab7731912bd2617d P 3ac7ff796e034848b5e46a3bc03d1c62 [LEADER]: Connected to new peer: Peer: permanent_uuid: "caf7bbda41c44c57b48367e7eda0d776" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 37789 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20260620 14:48:48.322727 11073 tablet.cc:2369] T 1ef26d2ae62044a7ab7731912bd2617d P 3ac7ff796e034848b5e46a3bc03d1c62: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260620 14:48:48.338625 10747 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:36398:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 14:48:48.338896 10747 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:36398:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 14:48:48.342473 10747 catalog_manager.cc:5782] T 00000000000000000000000000000000 P e87c3c7c892b42da8845070c0e2d98f8: Sending DeleteTablet for 3 replicas of tablet 1ef26d2ae62044a7ab7731912bd2617d
I20260620 14:48:48.343791 11141 tablet_service.cc:1514] Processing DeleteTablet for tablet 1ef26d2ae62044a7ab7731912bd2617d with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 14:48:48 UTC) from {username='kudu', principal='kudu/127.0.97.62@KRBTEST.COM'} at 127.0.0.1:45338
I20260620 14:48:48.343842 10873 tablet_service.cc:1514] Processing DeleteTablet for tablet 1ef26d2ae62044a7ab7731912bd2617d with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 14:48:48 UTC) from {username='kudu', principal='kudu/127.0.97.62@KRBTEST.COM'} at 127.0.0.1:34954
I20260620 14:48:48.343994 11007 tablet_service.cc:1514] Processing DeleteTablet for tablet 1ef26d2ae62044a7ab7731912bd2617d with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 14:48:48 UTC) from {username='kudu', principal='kudu/127.0.97.62@KRBTEST.COM'} at 127.0.0.1:46762
I20260620 14:48:48.345335 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskUci3DU/build/asan/bin/kudu pid 10810
I20260620 14:48:48.354826 11249 tablet_replica.cc:330] T 1ef26d2ae62044a7ab7731912bd2617d P caf7bbda41c44c57b48367e7eda0d776: stopping tablet replica
I20260620 14:48:48.356052 11250 tablet_replica.cc:330] T 1ef26d2ae62044a7ab7731912bd2617d P 3ac7ff796e034848b5e46a3bc03d1c62: stopping tablet replica
I20260620 14:48:48.356223 11249 raft_consensus.cc:2229] T 1ef26d2ae62044a7ab7731912bd2617d P caf7bbda41c44c57b48367e7eda0d776 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 14:48:48.356806 11250 raft_consensus.cc:2229] T 1ef26d2ae62044a7ab7731912bd2617d P 3ac7ff796e034848b5e46a3bc03d1c62 [term 1 LEADER]: Raft consensus shutting down.
I20260620 14:48:48.356904 11249 raft_consensus.cc:2258] T 1ef26d2ae62044a7ab7731912bd2617d P caf7bbda41c44c57b48367e7eda0d776 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 14:48:48.358029 11250 raft_consensus.cc:2258] T 1ef26d2ae62044a7ab7731912bd2617d P 3ac7ff796e034848b5e46a3bc03d1c62 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 14:48:48.359970 11249 ts_tablet_manager.cc:1905] T 1ef26d2ae62044a7ab7731912bd2617d P caf7bbda41c44c57b48367e7eda0d776: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 14:48:48.364183 11248 tablet_replica.cc:330] T 1ef26d2ae62044a7ab7731912bd2617d P 380b5d5555c244899f8a06998284b5a0: stopping tablet replica
I20260620 14:48:48.365341 11250 ts_tablet_manager.cc:1905] T 1ef26d2ae62044a7ab7731912bd2617d P 3ac7ff796e034848b5e46a3bc03d1c62: Deleting tablet data with delete state TABLET_DATA_DELETED
Jun 20 14:48:48 dist-test-slave-q9j6 krb5kdc[10703](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781966925, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.1@KRBTEST.COM
I20260620 14:48:48.368073 11248 raft_consensus.cc:2229] T 1ef26d2ae62044a7ab7731912bd2617d P 380b5d5555c244899f8a06998284b5a0 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 14:48:48.368595 11249 ts_tablet_manager.cc:1918] T 1ef26d2ae62044a7ab7731912bd2617d P caf7bbda41c44c57b48367e7eda0d776: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260620 14:48:48.368760 11249 log.cc:1196] T 1ef26d2ae62044a7ab7731912bd2617d P caf7bbda41c44c57b48367e7eda0d776: Deleting WAL directory at /tmp/dist-test-taskUci3DU/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781966829091300-388-0/minicluster-data/ts-2/wal/wals/1ef26d2ae62044a7ab7731912bd2617d
I20260620 14:48:48.369309 11249 ts_tablet_manager.cc:1939] T 1ef26d2ae62044a7ab7731912bd2617d P caf7bbda41c44c57b48367e7eda0d776: Deleting consensus metadata
I20260620 14:48:48.369297 11248 raft_consensus.cc:2258] T 1ef26d2ae62044a7ab7731912bd2617d P 380b5d5555c244899f8a06998284b5a0 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 14:48:48.371289 10733 catalog_manager.cc:4841] TS caf7bbda41c44c57b48367e7eda0d776 (127.0.97.3:37789): tablet 1ef26d2ae62044a7ab7731912bd2617d (table test-table [id=2c3f74c91d9a49e19e9dc23dec76fc06]) successfully deleted
I20260620 14:48:48.372922 11250 ts_tablet_manager.cc:1918] T 1ef26d2ae62044a7ab7731912bd2617d P 3ac7ff796e034848b5e46a3bc03d1c62: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260620 14:48:48.373121 11250 log.cc:1196] T 1ef26d2ae62044a7ab7731912bd2617d P 3ac7ff796e034848b5e46a3bc03d1c62: Deleting WAL directory at /tmp/dist-test-taskUci3DU/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781966829091300-388-0/minicluster-data/ts-1/wal/wals/1ef26d2ae62044a7ab7731912bd2617d
I20260620 14:48:48.373526 11250 ts_tablet_manager.cc:1939] T 1ef26d2ae62044a7ab7731912bd2617d P 3ac7ff796e034848b5e46a3bc03d1c62: Deleting consensus metadata
I20260620 14:48:48.374938 10735 catalog_manager.cc:4841] TS 3ac7ff796e034848b5e46a3bc03d1c62 (127.0.97.2:40449): tablet 1ef26d2ae62044a7ab7731912bd2617d (table test-table [id=2c3f74c91d9a49e19e9dc23dec76fc06]) successfully deleted
I20260620 14:48:48.376236 11248 ts_tablet_manager.cc:1905] T 1ef26d2ae62044a7ab7731912bd2617d P 380b5d5555c244899f8a06998284b5a0: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 14:48:48.378082 11220 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:48:48.346421 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34966 (local address 127.0.97.1:45285)
0620 14:48:48.346620 (+ 199us) server_negotiation.cc:206] Beginning negotiation
0620 14:48:48.346629 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0620 14:48:48.355705 (+ 9076us) server_negotiation.cc:407] Connection header received
0620 14:48:48.356006 (+ 301us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:48:48.356015 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:48:48.356134 (+ 119us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:48:48.356233 (+ 99us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:48:48.358924 (+ 2691us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:48:48.360154 (+ 1230us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:48:48.362045 (+ 1891us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:48:48.362709 (+ 664us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:48:48.362807 (+ 98us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:48:48.370657 (+ 7850us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 14:48:48.370672 (+ 15us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 14:48:48.370690 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 14:48:48.370736 (+ 46us) server_negotiation.cc:922] calling sasl_server_start()
0620 14:48:48.374711 (+ 3975us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:48:48.375694 (+ 983us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:48:48.375706 (+ 12us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:48:48.375710 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:48:48.375880 (+ 170us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:48:48.376340 (+ 460us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:48:48.376345 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:48:48.376348 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:48:48.376797 (+ 449us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 14:48:48.377111 (+ 314us) server_negotiation.cc:1035] Waiting for connection context
0620 14:48:48.377433 (+ 322us) server_negotiation.cc:299] Negotiation successful
0620 14:48:48.377666 (+ 233us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":55}
I20260620 14:48:48.378816 10933 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 14:48:48.484620 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskUci3DU/build/asan/bin/kudu with pid 10810
I20260620 14:48:48.502321 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskUci3DU/build/asan/bin/kudu pid 10944
W20260620 14:48:48.502224 10735 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.0.97.1:45285 (error 108)
I20260620 14:48:48.505496 11221 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:48:48.504995 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:45285 (local address 127.0.0.1:34968)
0620 14:48:48.505190 (+ 195us) negotiation.cc:107] Waiting for socket to connect
0620 14:48:48.505306 (+ 116us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.1:45285: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":64}
W20260620 14:48:48.506644 10735 catalog_manager.cc:4584] TS 380b5d5555c244899f8a06998284b5a0 (127.0.97.1:45285): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 1ef26d2ae62044a7ab7731912bd2617d: Network error: Client connection negotiation failed: client connection to 127.0.97.1:45285: connect: Connection refused (error 111)
Jun 20 14:48:48 dist-test-slave-q9j6 krb5kdc[10703](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781966925, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.2@KRBTEST.COM
I20260620 14:48:48.519130 11224 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:48:48.503353 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46770 (local address 127.0.97.2:40449)
0620 14:48:48.503522 (+ 169us) server_negotiation.cc:206] Beginning negotiation
0620 14:48:48.503528 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 14:48:48.503548 (+ 20us) server_negotiation.cc:407] Connection header received
0620 14:48:48.503648 (+ 100us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:48:48.503654 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:48:48.503738 (+ 84us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:48:48.503847 (+ 109us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:48:48.505444 (+ 1597us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:48:48.506346 (+ 902us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:48:48.507921 (+ 1575us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:48:48.508366 (+ 445us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:48:48.508523 (+ 157us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:48:48.512896 (+ 4373us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 14:48:48.512908 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 14:48:48.512924 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 14:48:48.512966 (+ 42us) server_negotiation.cc:922] calling sasl_server_start()
0620 14:48:48.516320 (+ 3354us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:48:48.517143 (+ 823us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:48:48.517150 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:48:48.517152 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:48:48.517288 (+ 136us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:48:48.517676 (+ 388us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:48:48.517680 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:48:48.517682 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:48:48.518025 (+ 343us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 14:48:48.518275 (+ 250us) server_negotiation.cc:1035] Waiting for connection context
0620 14:48:48.518610 (+ 335us) server_negotiation.cc:299] Negotiation successful
0620 14:48:48.518781 (+ 171us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":59}
I20260620 14:48:48.519891 11067 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 14:48:48.560496 11221 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:48:48.560064 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:45285 (local address 127.0.0.1:34972)
0620 14:48:48.560264 (+ 200us) negotiation.cc:107] Waiting for socket to connect
0620 14:48:48.560355 (+ 91us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.1:45285: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":74}
I20260620 14:48:48.561542 11221 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:48:48.561212 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:45285 (local address 127.0.0.1:34982)
0620 14:48:48.561331 (+ 119us) negotiation.cc:107] Waiting for socket to connect
0620 14:48:48.561412 (+ 81us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.1:45285: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":43}
I20260620 14:48:48.621955 11221 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:48:48.621505 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:45285 (local address 127.0.0.1:34986)
0620 14:48:48.621713 (+ 208us) negotiation.cc:107] Waiting for socket to connect
0620 14:48:48.621802 (+ 89us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.1:45285: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":78}
I20260620 14:48:48.623077 11221 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:48:48.622622 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:45285 (local address 127.0.0.1:34996)
0620 14:48:48.622812 (+ 190us) negotiation.cc:107] Waiting for socket to connect
0620 14:48:48.622893 (+ 81us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.1:45285: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":93}
I20260620 14:48:48.637392 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskUci3DU/build/asan/bin/kudu with pid 10944
I20260620 14:48:48.658552 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskUci3DU/build/asan/bin/kudu pid 11078
Jun 20 14:48:48 dist-test-slave-q9j6 krb5kdc[10703](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781966925, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.3@KRBTEST.COM
I20260620 14:48:48.672685 11222 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:48:48.659264 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45344 (local address 127.0.97.3:37789)
0620 14:48:48.659461 (+ 197us) server_negotiation.cc:206] Beginning negotiation
0620 14:48:48.659465 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 14:48:48.659484 (+ 19us) server_negotiation.cc:407] Connection header received
0620 14:48:48.659700 (+ 216us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:48:48.659704 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:48:48.659767 (+ 63us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:48:48.659851 (+ 84us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:48:48.660990 (+ 1139us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:48:48.661744 (+ 754us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:48:48.663111 (+ 1367us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:48:48.663485 (+ 374us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:48:48.663576 (+ 91us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:48:48.667835 (+ 4259us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 14:48:48.667850 (+ 15us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 14:48:48.667865 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 14:48:48.667899 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0620 14:48:48.670360 (+ 2461us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:48:48.670966 (+ 606us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:48:48.670973 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:48:48.670976 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:48:48.671118 (+ 142us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:48:48.671413 (+ 295us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:48:48.671416 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:48:48.671418 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:48:48.671827 (+ 409us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 14:48:48.672055 (+ 228us) server_negotiation.cc:1035] Waiting for connection context
0620 14:48:48.672289 (+ 234us) server_negotiation.cc:299] Negotiation successful
0620 14:48:48.672447 (+ 158us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":77}
I20260620 14:48:48.673348 11201 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 14:48:48.703652 11221 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:48:48.703092 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:45285 (local address 127.0.0.1:35004)
0620 14:48:48.703328 (+ 236us) negotiation.cc:107] Waiting for socket to connect
0620 14:48:48.703453 (+ 125us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.1:45285: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":82}
I20260620 14:48:48.704749 11221 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:48:48.704322 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:45285 (local address 127.0.0.1:35012)
0620 14:48:48.704529 (+ 207us) negotiation.cc:107] Waiting for socket to connect
0620 14:48:48.704614 (+ 85us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.1:45285: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":93}
I20260620 14:48:48.762077 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskUci3DU/build/asan/bin/kudu with pid 11078
I20260620 14:48:48.779084 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskUci3DU/build/asan/bin/kudu pid 10716
I20260620 14:48:48.780344 10778 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 14:48:48.887176 11221 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:48:48.886572 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:45285 (local address 127.0.0.1:35016)
0620 14:48:48.886829 (+ 257us) negotiation.cc:107] Waiting for socket to connect
0620 14:48:48.886945 (+ 116us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.1:45285: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":49}
I20260620 14:48:48.887143 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskUci3DU/build/asan/bin/kudu with pid 10716
2026-06-20T14:48:48Z chronyd exiting
[ OK ] SecurityITest.TestRequireAuthenticationSecureCluster (4282 ms)
[ RUN ] SecurityITest.TestEncryptionWithKMSIntegration
Loading random data
Initializing database '/tmp/dist-test-taskUci3DU/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781966829091300-388-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 20 14:48:48 dist-test-slave-q9j6 krb5kdc[11259](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 20 14:48:48 dist-test-slave-q9j6 krb5kdc[11259](info): set up 2 sockets
Jun 20 14:48:48 dist-test-slave-q9j6 krb5kdc[11259](info): commencing operation
krb5kdc: starting...
W20260620 14:48:49.980635 388 mini_kdc.cc:121] Time spent starting KDC: real 1.051s user 0.000s sys 0.024s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskUci3DU/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781966829091300-388-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskUci3DU/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781966829091300-388-0/krb5kdc/test-user.keytab.
Jun 20 14:48:50 dist-test-slave-q9j6 krb5kdc[11259](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781966930, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-20T14:48:50Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T14:48:50Z Disabled control of system clock
WARNING: no policy specified for rangeradmin/127.0.97.20@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangeradmin/127.0.97.20@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangeradmin/127.0.97.20@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskUci3DU/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781966829091300-388-0/krb5kdc/rangeradmin_127.0.97.20@KRBTEST.COM.keytab.
Entry for principal rangeradmin/127.0.97.20@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskUci3DU/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781966829091300-388-0/krb5kdc/rangeradmin_127.0.97.20@KRBTEST.COM.keytab.
WARNING: no policy specified for rangerlookup/127.0.97.20@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangerlookup/127.0.97.20@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangerlookup/127.0.97.20@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskUci3DU/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781966829091300-388-0/krb5kdc/rangerlookup_127.0.97.20@KRBTEST.COM.keytab.
Entry for principal rangerlookup/127.0.97.20@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskUci3DU/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781966829091300-388-0/krb5kdc/rangerlookup_127.0.97.20@KRBTEST.COM.keytab.
WARNING: no policy specified for HTTP/127.0.97.20@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.0.97.20@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.0.97.20@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskUci3DU/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781966829091300-388-0/krb5kdc/HTTP_127.0.97.20@KRBTEST.COM.keytab.
Entry for principal HTTP/127.0.97.20@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskUci3DU/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781966829091300-388-0/krb5kdc/HTTP_127.0.97.20@KRBTEST.COM.keytab.
I20260620 14:48:50.312402 388 mini_postgres.cc:62] Running initdb...
The files belonging to this database system will be owned by user "slave".
This user must also own the server process.
The database cluster will be initialized with locale "C".
The default database encoding has accordingly been set to "SQL_ASCII".
The default text search configuration will be set to "english".
Data page checksums are disabled.
creating directory /tmp/dist-test-taskUci3DU/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781966829091300-388-0/minicluster-data/postgres ... ok
creating subdirectories ... ok
selecting dynamic shared memory implementation ... posix
selecting default max_connections ... 100
selecting default shared_buffers ... 128MB
selecting default time zone ... Etc/UTC
creating configuration files ... ok
running bootstrap script ... ok
performing post-bootstrap initialization ... ok
syncing data to disk ... ok
initdb: warning: enabling "trust" authentication for local connections
You can change this by editing pg_hba.conf or using the option -A, or
--auth-local and --auth-host, the next time you run initdb.
Success. You can now start the database server using:
/tmp/dist-test-taskUci3DU/build/asan/bin/postgres/pg_ctl -D /tmp/dist-test-taskUci3DU/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781966829091300-388-0/minicluster-data/postgres -l logfile start
2026-06-20 14:48:52.655 UTC [11289] LOG: starting PostgreSQL 12.2 on x86_64-pc-linux-gnu, compiled by cc (GCC) 8.3.1 20190311 (Red Hat 8.3.1-3), 64-bit
2026-06-20 14:48:52.655 UTC [11289] LOG: listening on IPv4 address "127.0.97.20", port 48703
2026-06-20 14:48:52.662 UTC [11289] LOG: listening on Unix socket "/tmp/.s.PGSQL.48703"
2026-06-20 14:48:52.684 UTC [11292] LOG: database system was shut down at 2026-06-20 14:48:51 UTC
2026-06-20 14:48:52.687 UTC [11289] LOG: database system is ready to accept connections
I20260620 14:48:53.660290 388 mini_postgres.cc:96] Postgres bound to 48703
2026-06-20 14:48:53.678 UTC [11300] FATAL: database "slave" does not exist
127.0.97.20:48703 - accepting connections
I20260620 14:48:53.679678 388 mini_ranger.cc:161] Starting Ranger...
I20260620 14:48:53.707273 388 mini_ranger.cc:84] Created miniranger Postgres user
I20260620 14:48:54.139297 388 mini_ranger.cc:87] Created ranger Postgres database
I20260620 14:48:54.139439 388 mini_ranger.cc:178] Starting Ranger out of /tmp/dist-test-taskUci3DU/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781966829091300-388-0/minicluster-data/ranger-admin
2026-06-20 14:48:54,427 [I] DB FLAVOR :POSTGRES
2026-06-20 14:48:54,427 [I] --------- Verifying Ranger DB connection ---------
2026-06-20 14:48:54,427 [I] Checking connection..
2026-06-20 14:48:54,427 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskUci3DU/build/asan/bin/postgresql.jar:/tmp/dist-test-taskUci3DU/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.97.20:48703/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select 1;"
2026-06-20 14:48:54,676 [I] Checking connection passed.
2026-06-20 14:48:54,677 [I] --------- Verifying version history table ---------
2026-06-20 14:48:54,677 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskUci3DU/build/asan/bin/postgresql.jar:/tmp/dist-test-taskUci3DU/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.97.20:48703/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-06-20 14:48:54,931 [I] Table x_db_version_h does not exist in database ranger
2026-06-20 14:48:54,932 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskUci3DU/build/asan/bin/postgresql.jar:/tmp/dist-test-taskUci3DU/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.97.20:48703/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-06-20 14:48:55,162 [I] Table x_db_version_h does not exist in database ranger
2026-06-20 14:48:55,162 [I] Importing x_db_version_h table schema to database ranger from file: create_dbversion_catalog.sql
2026-06-20 14:48:55,163 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskUci3DU/build/asan/bin/postgresql.jar:/tmp/dist-test-taskUci3DU/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.97.20:48703/ranger -u miniranger -p '********' -noheader -trim -c \; -input /tmp/dist-test-taskUci3DU/thirdparty/src/ranger-2.1.0-admin/db/postgres/create_dbversion_catalog.sql
2026-06-20 14:48:55.399 UTC [11364] WARNING: there is no transaction in progress
2026-06-20 14:48:55,405 [I] create_dbversion_catalog.sql file imported successfully
2026-06-20 14:48:55,406 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskUci3DU/build/asan/bin/postgresql.jar:/tmp/dist-test-taskUci3DU/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.97.20:48703/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-06-20 14:48:55,660 [I] Table x_db_version_h already exists in database 'ranger'
2026-06-20 14:48:55,660 [I] --------- Importing Ranger Core DB Schema ---------
2026-06-20 14:48:55,661 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskUci3DU/build/asan/bin/postgresql.jar:/tmp/dist-test-taskUci3DU/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.97.20:48703/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'CORE_DB_SCHEMA' and active = 'Y';"
2026-06-20 14:48:55,912 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskUci3DU/build/asan/bin/postgresql.jar:/tmp/dist-test-taskUci3DU/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.97.20:48703/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'CORE_DB_SCHEMA' and active = 'N';"
2026-06-20 14:48:56,182 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskUci3DU/build/asan/bin/postgresql.jar:/tmp/dist-test-taskUci3DU/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.97.20:48703/ranger -u miniranger -p '********' -noheader -trim -c \; -query "insert into x_db_version_h (version, inst_at, inst_by, updated_at, updated_by,active) values ('CORE_DB_SCHEMA', current_timestamp, 'Ranger 2.1.0', current_timestamp, 'dist-test-slave-q9j6.c.gcp-upstream.internal','N') ;"
2026-06-20 14:48:56,423 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskUci3DU/build/asan/bin/postgresql.jar:/tmp/dist-test-taskUci3DU/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.97.20:48703/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_portal_user') as temp;"
2026-06-20 14:48:56,688 [I] Table x_portal_user does not exist in database ranger
2026-06-20 14:48:56,688 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskUci3DU/build/asan/bin/postgresql.jar:/tmp/dist-test-taskUci3DU/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.97.20:48703/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_policy_ref_group') as temp;"
2026-06-20 14:48:56,923 [I] Table x_policy_ref_group does not exist in database ranger
2026-06-20 14:48:56,924 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskUci3DU/build/asan/bin/postgresql.jar:/tmp/dist-test-taskUci3DU/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.97.20:48703/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and active = 'Y';"
2026-06-20 14:48:57,154 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskUci3DU/build/asan/bin/postgresql.jar:/tmp/dist-test-taskUci3DU/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.97.20:48703/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'JAVA_PATCHES' and active = 'Y';"
2026-06-20 14:48:57,383 [I] Importing DB schema to database ranger from file: ranger_core_db_postgres.sql
2026-06-20 14:48:57,383 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskUci3DU/build/asan/bin/postgresql.jar:/tmp/dist-test-taskUci3DU/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.97.20:48703/ranger -u miniranger -p '********' -noheader -trim -c \; -input /tmp/dist-test-taskUci3DU/thirdparty/src/ranger-2.1.0-admin/db/postgres/optimized/current/ranger_core_db_postgres.sql
2026-06-20 14:48:58.322 UTC [11472] WARNING: there is no transaction in progress
2026-06-20 14:48:58.339 UTC [11472] WARNING: there is no transaction in progress
2026-06-20 14:48:58.355 UTC [11472] WARNING: there is no transaction in progress
2026-06-20 14:48:58.370 UTC [11472] WARNING: there is no transaction in progress
2026-06-20 14:48:58.384 UTC [11472] WARNING: there is no transaction in progress
2026-06-20 14:48:58.401 UTC [11472] WARNING: there is no transaction in progress
2026-06-20 14:48:58.466 UTC [11472] WARNING: there is no transaction in progress
2026-06-20 14:48:58.472 UTC [11472] WARNING: there is no transaction in progress
2026-06-20 14:48:58.487 UTC [11472] WARNING: there is no transaction in progress
2026-06-20 14:48:58.497 UTC [11472] WARNING: there is no transaction in progress
2026-06-20 14:48:58.507 UTC [11472] WARNING: there is no transaction in progress
2026-06-20 14:48:58.520 UTC [11472] WARNING: there is no transaction in progress
2026-06-20 14:48:58.529 UTC [11472] WARNING: there is no transaction in progress
2026-06-20 14:48:58.535 UTC [11472] WARNING: there is no transaction in progress
2026-06-20 14:48:58.542 UTC [11472] WARNING: there is no transaction in progress
2026-06-20 14:48:58.548 UTC [11472] WARNING: there is no transaction in progress
2026-06-20 14:48:59,156 [I] ranger_core_db_postgres.sql file imported successfully
2026-06-20 14:48:59,157 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskUci3DU/build/asan/bin/postgresql.jar:/tmp/dist-test-taskUci3DU/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.97.20:48703/ranger -u miniranger -p '********' -noheader -trim -c \; -query "update x_db_version_h set inst_by='Ranger 2.1.0' where active='Y' and updated_by='localhost';"
2026-06-20 14:48:59,407 [I] Patches status entries updated from base ranger version to current installed ranger version:Ranger 2.1.0
2026-06-20 14:48:59,407 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskUci3DU/build/asan/bin/postgresql.jar:/tmp/dist-test-taskUci3DU/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.97.20:48703/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_portal_user') as temp;"
2026-06-20 14:48:59,668 [I] Table x_portal_user already exists in database 'ranger'
2026-06-20 14:48:59,669 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskUci3DU/build/asan/bin/postgresql.jar:/tmp/dist-test-taskUci3DU/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.97.20:48703/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_policy_ref_group') as temp;"
2026-06-20 14:48:59,911 [I] Table x_policy_ref_group already exists in database 'ranger'
2026-06-20 14:48:59,912 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskUci3DU/build/asan/bin/postgresql.jar:/tmp/dist-test-taskUci3DU/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.97.20:48703/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and active = 'Y';"
2026-06-20 14:49:00,147 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskUci3DU/build/asan/bin/postgresql.jar:/tmp/dist-test-taskUci3DU/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.97.20:48703/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'JAVA_PATCHES' and active = 'Y';"
2026-06-20 14:49:00,409 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskUci3DU/build/asan/bin/postgresql.jar:/tmp/dist-test-taskUci3DU/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.97.20:48703/ranger -u miniranger -p '********' -noheader -trim -c \; -query "update x_db_version_h set active='Y' where version='CORE_DB_SCHEMA' and active='N' and updated_by='dist-test-slave-q9j6.c.gcp-upstream.internal';"
2026-06-20 14:49:00,636 [I] CORE_DB_SCHEMA import status has been updated
2026-06-20 14:49:00,637 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskUci3DU/build/asan/bin/postgresql.jar:/tmp/dist-test-taskUci3DU/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.97.20:48703/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and inst_by = 'Ranger 2.1.0' and active = 'Y';"
2026-06-20 14:49:00,892 [I] DB_PATCHES have already been applied
I20260620 14:49:00.898831 388 mini_ranger.cc:191] Using Ranger class path: /tmp/dist-test-taskUci3DU/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781966829091300-388-0/minicluster-data/ranger-admin:/tmp/dist-test-taskUci3DU/thirdparty/src/ranger-2.1.0-admin/ews/lib/*:/usr/lib/jvm/java-1.8.0-openjdk-amd64/lib/*:/tmp/dist-test-taskUci3DU/thirdparty/src/hadoop-3.2.0/*:/tmp/dist-test-taskUci3DU/build/asan/bin/postgresql.jar:/tmp/dist-test-taskUci3DU/thirdparty/src/ranger-2.1.0-admin/ews/webapp
I20260620 14:49:00.899003 388 mini_ranger.cc:193] Using host: 127.0.97.20
I20260620 14:49:00.914427 388 mini_ranger.cc:227] Ranger admin URL: http://127.0.97.20:50299
Jun 20, 2026 2:49:01 PM org.apache.ranger.server.tomcat.EmbeddedServer getKeyManagers
WARNING: Config 'ranger.keystore.file' or 'ranger.service.https.attrib.keystore.file' is not found or contains blank value
Jun 20, 2026 2:49:01 PM org.apache.ranger.server.tomcat.EmbeddedServer getTrustManagers
WARNING: Config 'ranger.truststore.file' is not found or contains blank value!
Jun 20, 2026 2:49:01 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Deriving webapp folder from catalina.base property. folder=/tmp/dist-test-taskUci3DU/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781966829091300-388-0/minicluster-data/ranger-admin/ews/webapp
Jun 20, 2026 2:49:01 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Webapp file =/tmp/dist-test-taskUci3DU/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781966829091300-388-0/minicluster-data/ranger-admin/ews/webapp, webAppName = /
Jun 20, 2026 2:49:01 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Adding webapp [/] = path [/tmp/dist-test-taskUci3DU/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781966829091300-388-0/minicluster-data/ranger-admin/ews/webapp] .....
Jun 20, 2026 2:49:01 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Finished init of webapp [/] = path [/tmp/dist-test-taskUci3DU/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781966829091300-388-0/minicluster-data/ranger-admin/ews/webapp].
Jun 20, 2026 2:49:01 PM org.apache.ranger.server.tomcat.EmbeddedServer startServer
INFO: Server Name : miniranger
I20260620 14:49:02.050980 388 mini_ranger.cc:160] Time spent starting Ranger: real 8.371s user 0.004s sys 0.037s
Jun 20, 2026 2:49:21 PM com.sun.jersey.api.core.PackagesResourceConfig init
INFO: Scanning for root resource and provider classes in the packages:
org.apache.ranger.rest
org.apache.ranger.common
xa.rest
Jun 20, 2026 2:49:21 PM com.sun.jersey.api.core.ScanningResourceConfig logClasses
INFO: Root resource classes found:
class org.apache.ranger.rest.MetricsREST
class org.apache.ranger.rest.XKeyREST
class org.apache.ranger.rest.XAuditREST
class org.apache.ranger.rest.XUserREST
class org.apache.ranger.rest.UserREST
class org.apache.ranger.rest.ServiceREST
class org.apache.ranger.rest.PublicAPIs
class org.apache.ranger.rest.TagREST
class org.apache.ranger.rest.AssetREST
class org.apache.ranger.rest.RoleREST
class org.apache.ranger.rest.SecurityZoneREST
class org.apache.ranger.rest.PublicAPIsv2
Jun 20, 2026 2:49:21 PM com.sun.jersey.api.core.ScanningResourceConfig logClasses
INFO: Provider classes found:
class org.apache.ranger.common.RangerJAXBContextResolver
Jun 20, 2026 2:49:21 PM com.sun.jersey.spi.spring.container.servlet.SpringServlet getContext
INFO: Using default applicationContext
Jun 20, 2026 2:49:21 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, assetREST, of type org.apache.ranger.rest.AssetREST as a root resource class
Jun 20, 2026 2:49:21 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, metricsREST, of type org.apache.ranger.rest.MetricsREST as a root resource class
Jun 20, 2026 2:49:21 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, publicAPIs, of type org.apache.ranger.rest.PublicAPIs as a root resource class
Jun 20, 2026 2:49:21 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, publicAPIsv2, of type org.apache.ranger.rest.PublicAPIsv2 as a root resource class
Jun 20, 2026 2:49:21 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, roleREST, of type org.apache.ranger.rest.RoleREST as a root resource class
Jun 20, 2026 2:49:21 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, securityZoneREST, of type org.apache.ranger.rest.SecurityZoneREST as a root resource class
Jun 20, 2026 2:49:21 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, serviceREST, of type org.apache.ranger.rest.ServiceREST as a root resource class
Jun 20, 2026 2:49:21 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, tagREST, of type org.apache.ranger.rest.TagREST as a root resource class
Jun 20, 2026 2:49:21 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, userREST, of type org.apache.ranger.rest.UserREST as a root resource class
Jun 20, 2026 2:49:21 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XAuditREST, of type org.apache.ranger.rest.XAuditREST as a root resource class
Jun 20, 2026 2:49:21 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XKeyREST, of type org.apache.ranger.rest.XKeyREST as a root resource class
Jun 20, 2026 2:49:21 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XUserREST, of type org.apache.ranger.rest.XUserREST as a root resource class
Jun 20, 2026 2:49:21 PM com.sun.jersey.server.impl.application.WebApplicationImpl _initiate
INFO: Initiating Jersey application, version 'Jersey: 1.19.3 10/24/2016 03:58 PM'
Jun 20, 2026 2:49:22 PM com.sun.jersey.spi.inject.Errors processErrorMessages
WARNING: The following warnings have been detected with resource and/or provider classes:
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInExcel(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse), MUST return a non-void type.
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInJson(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse,java.lang.Boolean), MUST return a non-void type.
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInCsv(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse) throws java.io.IOException, MUST return a non-void type.
I20260620 14:49:23.262259 388 mini_ranger.cc:261] Created Kudu service
WARNING: no policy specified for rangerkms/127.0.97.20@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangerkms/127.0.97.20@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangerkms/127.0.97.20@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskUci3DU/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781966829091300-388-0/krb5kdc/rangerkms_127.0.97.20@KRBTEST.COM.keytab.
Entry for principal rangerkms/127.0.97.20@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskUci3DU/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781966829091300-388-0/krb5kdc/rangerkms_127.0.97.20@KRBTEST.COM.keytab.
WARNING: no policy specified for HTTP/127.0.97.20@KRBTEST.COM; defaulting to no policy
add_principal: Principal or policy already exists while creating "HTTP/127.0.97.20@KRBTEST.COM".
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.0.97.20@KRBTEST.COM with kvno 3, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskUci3DU/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781966829091300-388-0/krb5kdc/HTTP_127.0.97.20@KRBTEST.COM.keytab.
Entry for principal HTTP/127.0.97.20@KRBTEST.COM with kvno 3, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskUci3DU/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781966829091300-388-0/krb5kdc/HTTP_127.0.97.20@KRBTEST.COM.keytab.
WARNING: no policy specified for keyadmin@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "keyadmin@KRBTEST.COM" created.
Jun 20 14:49:23 dist-test-slave-q9j6 krb5kdc[11259](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781966963, etypes {rep=17 tkt=17 ses=17}, keyadmin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for keyadmin@KRBTEST.COM:
I20260620 14:49:23.431049 388 mini_ranger_kms.cc:208] Starting Ranger KMS...
I20260620 14:49:23.453274 388 mini_ranger_kms.cc:78] Created minirangerkms Postgres user
I20260620 14:49:24.259728 388 mini_ranger_kms.cc:81] Created rangerkms Postgres database
I20260620 14:49:24.259873 388 mini_ranger_kms.cc:226] Starting Ranger KMS out of /tmp/dist-test-taskUci3DU/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781966829091300-388-0/minicluster-data/ranger-kms
I20260620 14:49:24.260025 388 mini_ranger_kms.cc:227] Using postgres at 127.0.97.20:48703
2026-06-20 14:49:24,687 [I] DB FLAVOR :POSTGRES
2026-06-20 14:49:24,688 [I] --------- Verifying Ranger DB connection ---------
2026-06-20 14:49:24,688 [I] Checking connection
2026-06-20 14:49:24,688 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskUci3DU/build/asan/bin/postgresql.jar:/tmp/dist-test-taskUci3DU/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781966829091300-388-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.97.20:48703/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -query "SELECT 1;"
2026-06-20 14:49:24,925 [I] connection success
2026-06-20 14:49:24,925 [I] --------- Verifying Ranger DB tables ---------
2026-06-20 14:49:24,925 [I] Verifying table ranger_masterkey in database rangerkms
2026-06-20 14:49:24,925 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskUci3DU/build/asan/bin/postgresql.jar:/tmp/dist-test-taskUci3DU/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781966829091300-388-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.97.20:48703/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='rangerkms' and table_name = 'ranger_masterkey') as temp;"
2026-06-20 14:49:25,193 [I] Table ranger_masterkey does not exist in database rangerkms
2026-06-20 14:49:25,194 [I] --------- Importing Ranger Core DB Schema ---------
2026-06-20 14:49:25,194 [I] Importing db schema to database rangerkms from file: kms_core_db_postgres.sql
2026-06-20 14:49:25,194 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskUci3DU/build/asan/bin/postgresql.jar:/tmp/dist-test-taskUci3DU/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781966829091300-388-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.97.20:48703/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -input /tmp/dist-test-taskUci3DU/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781966829091300-388-0/minicluster-data/ranger-kms/db/postgres/kms_core_db_postgres.sql
2026-06-20 14:49:25,471 [I] kms_core_db_postgres.sql DB schema imported successfully
I20260620 14:49:25.969029 388 mini_ranger_kms.cc:310] Created kms service
I20260620 14:49:26.191434 388 mini_ranger_kms.cc:326] Created kudu user
I20260620 14:49:26.237268 388 mini_ranger_kms.cc:343] Created rangerkms user
Jun 20 14:49:26 dist-test-slave-q9j6 krb5kdc[11259](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781966963, etypes {rep=17 tkt=17 ses=17}, keyadmin@KRBTEST.COM for HTTP/127.0.97.20@KRBTEST.COM
I20260620 14:49:26.655275 388 mini_ranger_kms.cc:384] Added ranger policy
I20260620 14:49:26.655649 388 mini_ranger_kms.cc:238] Using RangerKMS classpath: /tmp/dist-test-taskUci3DU/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781966829091300-388-0/minicluster-data/ranger-kms:/tmp/dist-test-taskUci3DU/build/asan/bin/postgresql.jar:/tmp/dist-test-taskUci3DU/thirdparty/src/ranger-2.1.0-kms/ews/webapp/WEB-INF/classes/lib/*:/tmp/dist-test-taskUci3DU/thirdparty/src/ranger-2.1.0-kms/ews/webapp/lib/*:/usr/lib/jvm/java-1.8.0-openjdk-amd64/lib/*
I20260620 14:49:26.655715 388 mini_ranger_kms.cc:240] Using host: 127.0.97.20
I20260620 14:49:26.676522 388 mini_ranger_kms.cc:276] Ranger KMS PID: 11652
I20260620 14:49:26.676726 388 mini_ranger_kms.cc:277] Ranger KMS URL: http://127.0.97.20:52961
Jun 20, 2026 2:49:27 PM org.apache.ranger.server.tomcat.EmbeddedServer getKeyManagers
WARNING: Config 'ranger.keystore.file' or 'ranger.service.https.attrib.keystore.file' is not found or contains blank value
Jun 20, 2026 2:49:27 PM org.apache.ranger.server.tomcat.EmbeddedServer getTrustManagers
WARNING: Config 'ranger.truststore.file' is not found or contains blank value!
Jun 20, 2026 2:49:27 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Webapp file =/tmp/dist-test-taskUci3DU/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781966829091300-388-0/minicluster-data/ranger-kms/ews/webapp, webAppName = /kms
Jun 20, 2026 2:49:27 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Adding webapp [/kms] = path [/tmp/dist-test-taskUci3DU/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781966829091300-388-0/minicluster-data/ranger-kms/ews/webapp] .....
Jun 20, 2026 2:49:28 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Finished init of webapp [/kms] = path [/tmp/dist-test-taskUci3DU/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781966829091300-388-0/minicluster-data/ranger-kms/ews/webapp].
Jun 20, 2026 2:49:28 PM org.apache.ranger.server.tomcat.EmbeddedServer startServer
INFO: Server Name : minirangerkms
Jun 20 14:49:28 dist-test-slave-q9j6 krb5kdc[11259](info): AS_REQ (1 etypes {17}) 127.0.0.1: ISSUE: authtime 1781966968, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.0.97.20@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 14:49:29.012360 388 mini_ranger_kms.cc:207] Time spent starting Ranger KMS: real 5.581s user 0.115s sys 0.337s
I20260620 14:49:29.012629 388 mini_ranger_kms.cc:397] {"name":"kuduclusterkey","cipher":"AES/CTR/NoPadding","length":128,"description":"kuduclusterkey"}
I20260620 14:49:29.012840 388 mini_ranger_kms.cc:401] 127.0.97.20:52961/kms/v1/keys
Jun 20 14:49:29 dist-test-slave-q9j6 krb5kdc[11259](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781966968, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.0.97.20@KRBTEST.COM for HTTP/127.0.97.20@KRBTEST.COM
Jun 20 14:49:29 dist-test-slave-q9j6 krb5kdc[11259](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781966968, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.0.97.20@KRBTEST.COM for HTTP/127.0.97.20@KRBTEST.COM
Jun 20 14:49:31 dist-test-slave-q9j6 krb5kdc[11259](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781966971, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
WARNING: no policy specified for kudu/127.0.97.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskUci3DU/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781966829091300-388-0/krb5kdc/kudu_127.0.97.62.keytab.
Entry for principal kudu/127.0.97.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskUci3DU/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781966829091300-388-0/krb5kdc/kudu_127.0.97.62.keytab.
I20260620 14:49:32.027881 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskUci3DU/build/asan/bin/kudu
/tmp/dist-test-taskUci3DU/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskUci3DU/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781966829091300-388-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskUci3DU/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781966829091300-388-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskUci3DU/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781966829091300-388-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskUci3DU/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781966829091300-388-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:37829
--webserver_interface=127.0.97.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskUci3DU/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781966829091300-388-0/krb5kdc/kudu_127.0.97.62.keytab
--principal=kudu/127.0.97.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:41959
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.97.62:37829
--ranger_config_path=/tmp/dist-test-taskUci3DU/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781966829091300-388-0/minicluster-data/ranger-client
--trusted_user_acl=test-admin
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.0.97.20:52961/kms
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskUci3DU/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781966829091300-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskUci3DU/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781966829091300-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskUci3DU/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781966829091300-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 14:49:32.379436 11703 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 14:49:32.379827 11703 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 14:49:32.380018 11703 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 14:49:32.387811 11703 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 14:49:32.387929 11703 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 14:49:32.387986 11703 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 14:49:32.388098 11703 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 14:49:32.388186 11703 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 14:49:32.398573 11703 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:41959
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskUci3DU/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781966829091300-388-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskUci3DU/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781966829091300-388-0/minicluster-data/master-0/wal
--ranger_kms_url=127.0.97.20:52961/kms
--trusted_user_acl=<redacted>
--ipki_ca_key_size=768
--master_addresses=127.0.97.62:37829
--ranger_config_path=/tmp/dist-test-taskUci3DU/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781966829091300-388-0/minicluster-data/ranger-client
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskUci3DU/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781966829091300-388-0/krb5kdc/kudu_127.0.97.62.keytab
--principal=kudu/127.0.97.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:37829
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskUci3DU/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781966829091300-388-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.97.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskUci3DU/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781966829091300-388-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 13:41:20 UTC on ccycloud.quasar-vdagoi.root.comops.site
build id 4457462
ASAN enabled
I20260620 14:49:32.400003 11703 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 14:49:32.401396 11703 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 14:49:32.415410 11709 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 14:49:32.415663 11708 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 14:49:32.415686 11711 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 14:49:32.416606 11703 server_base.cc:1034] running on GCE node
I20260620 14:49:32.417433 11703 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 14:49:32.418984 11703 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 14:49:32.420240 11703 hybrid_clock.cc:648] HybridClock initialized: now 1781966972420208 us; error 58 us; skew 500 ppm
I20260620 14:49:32.420631 11703 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 14:49:32 dist-test-slave-q9j6 krb5kdc[11259](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781966972, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 14:49:32.427839 11703 init.cc:377] Logged in from keytab as kudu/127.0.97.62@KRBTEST.COM (short username kudu)
I20260620 14:49:32.429543 11703 webserver.cc:460] Webserver started at http://127.0.97.62:44803/ using document root <none> and password file <none>
I20260620 14:49:32.430301 11703 fs_manager.cc:359] Metadata directory not provided
I20260620 14:49:32.430459 11703 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 14:49:32.430783 11703 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Jun 20 14:49:32 dist-test-slave-q9j6 krb5kdc[11259](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781966972, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.62@KRBTEST.COM for HTTP/127.0.97.20@KRBTEST.COM
I20260620 14:49:32.482497 11703 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskUci3DU/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781966829091300-388-0/minicluster-data/master-0/data/instance:
uuid: "68faae2f3b5c445c898441cb1466402e"
format_stamp: "Formatted at 2026-06-20 14:49:32 on dist-test-slave-q9j6"
server_key: "b746522736ecd512a1090c1140f6f4cf"
server_key_iv: "688e98f290a0585853924fccd68df8d5"
server_key_version: "kuduclusterkey@0"
I20260620 14:49:32.483702 11703 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskUci3DU/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781966829091300-388-0/minicluster-data/master-0/wal/instance:
uuid: "68faae2f3b5c445c898441cb1466402e"
format_stamp: "Formatted at 2026-06-20 14:49:32 on dist-test-slave-q9j6"
server_key: "b746522736ecd512a1090c1140f6f4cf"
server_key_iv: "688e98f290a0585853924fccd68df8d5"
server_key_version: "kuduclusterkey@0"
I20260620 14:49:32.489743 11703 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.005s sys 0.000s
I20260620 14:49:32.510637 11718 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 14:49:32.512579 11703 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.003s sys 0.000s
I20260620 14:49:32.512791 11703 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskUci3DU/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781966829091300-388-0/minicluster-data/master-0/data,/tmp/dist-test-taskUci3DU/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781966829091300-388-0/minicluster-data/master-0/wal
uuid: "68faae2f3b5c445c898441cb1466402e"
format_stamp: "Formatted at 2026-06-20 14:49:32 on dist-test-slave-q9j6"
server_key: "b746522736ecd512a1090c1140f6f4cf"
server_key_iv: "688e98f290a0585853924fccd68df8d5"
server_key_version: "kuduclusterkey@0"
I20260620 14:49:32.513072 11703 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskUci3DU/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781966829091300-388-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskUci3DU/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781966829091300-388-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskUci3DU/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781966829091300-388-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 14:49:32.553378 11703 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 14:49:32.557184 11703 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 14:49:32.557430 11703 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 14:49:32.578092 11703 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.62:37829
I20260620 14:49:32.578126 11769 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.62:37829 every 8 connection(s)
I20260620 14:49:32.579536 11703 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskUci3DU/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781966829091300-388-0/minicluster-data/master-0/data/info.pb
I20260620 14:49:32.583361 11770 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 14:49:32.589105 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskUci3DU/build/asan/bin/kudu as pid 11703
I20260620 14:49:32.589323 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskUci3DU/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781966829091300-388-0/minicluster-data/master-0/wal/instance
I20260620 14:49:32.589957 388 external_mini_cluster.cc:1428] Setting key 9d6c780d1cc6ff388b23263b6adcdee5
I20260620 14:49:32.596016 11770 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 68faae2f3b5c445c898441cb1466402e: Bootstrap starting.
I20260620 14:49:32.600059 11770 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 68faae2f3b5c445c898441cb1466402e: Neither blocks nor log segments found. Creating new log.
I20260620 14:49:32.601168 11770 log.cc:824] T 00000000000000000000000000000000 P 68faae2f3b5c445c898441cb1466402e: Log is configured to *not* fsync() on all Append() calls
I20260620 14:49:32.604835 11770 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 68faae2f3b5c445c898441cb1466402e: No bootstrap required, opened a new log
Jun 20 14:49:32 dist-test-slave-q9j6 krb5kdc[11259](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781966971, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260620 14:49:32.613567 11770 raft_consensus.cc:348] T 00000000000000000000000000000000 P 68faae2f3b5c445c898441cb1466402e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "68faae2f3b5c445c898441cb1466402e" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 37829 } }
I20260620 14:49:32.614097 11770 raft_consensus.cc:374] T 00000000000000000000000000000000 P 68faae2f3b5c445c898441cb1466402e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 14:49:32.614235 11770 raft_consensus.cc:729] T 00000000000000000000000000000000 P 68faae2f3b5c445c898441cb1466402e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 68faae2f3b5c445c898441cb1466402e, State: Initialized, Role: FOLLOWER
I20260620 14:49:32.615047 11770 consensus_queue.cc:260] T 00000000000000000000000000000000 P 68faae2f3b5c445c898441cb1466402e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "68faae2f3b5c445c898441cb1466402e" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 37829 } }
I20260620 14:49:32.615391 11770 raft_consensus.cc:388] T 00000000000000000000000000000000 P 68faae2f3b5c445c898441cb1466402e [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 14:49:32.615363 11773 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:49:32.592218 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38274 (local address 127.0.97.62:37829)
0620 14:49:32.593187 (+ 969us) server_negotiation.cc:206] Beginning negotiation
0620 14:49:32.593205 (+ 18us) server_negotiation.cc:399] Waiting for connection header
0620 14:49:32.593260 (+ 55us) server_negotiation.cc:407] Connection header received
0620 14:49:32.595622 (+ 2362us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:49:32.595656 (+ 34us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:49:32.596197 (+ 541us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:49:32.596672 (+ 475us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:49:32.597836 (+ 1164us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:49:32.599875 (+ 2039us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:49:32.601271 (+ 1396us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:49:32.602315 (+ 1044us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:49:32.602558 (+ 243us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:49:32.606519 (+ 3961us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 14:49:32.606528 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 14:49:32.606544 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 14:49:32.606583 (+ 39us) server_negotiation.cc:922] calling sasl_server_start()
0620 14:49:32.610122 (+ 3539us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:49:32.610937 (+ 815us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:49:32.610945 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:49:32.610952 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:49:32.611103 (+ 151us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:49:32.611582 (+ 479us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:49:32.611588 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:49:32.611591 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:49:32.612246 (+ 655us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 14:49:32.612587 (+ 341us) server_negotiation.cc:1035] Waiting for connection context
0620 14:49:32.613145 (+ 558us) server_negotiation.cc:299] Negotiation successful
0620 14:49:32.613628 (+ 483us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":681,"thread_start_us":437,"threads_started":1}
I20260620 14:49:32.615624 11770 raft_consensus.cc:482] T 00000000000000000000000000000000 P 68faae2f3b5c445c898441cb1466402e [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 14:49:32.615864 11770 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 68faae2f3b5c445c898441cb1466402e [term 0 FOLLOWER]: Advancing to term 1
I20260620 14:49:32.619925 11770 raft_consensus.cc:504] T 00000000000000000000000000000000 P 68faae2f3b5c445c898441cb1466402e [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "68faae2f3b5c445c898441cb1466402e" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 37829 } }
I20260620 14:49:32.620647 11770 leader_election.cc:302] T 00000000000000000000000000000000 P 68faae2f3b5c445c898441cb1466402e [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 68faae2f3b5c445c898441cb1466402e; no voters:
I20260620 14:49:32.621379 11770 leader_election.cc:288] T 00000000000000000000000000000000 P 68faae2f3b5c445c898441cb1466402e [CANDIDATE]: Term 1 election: Requested vote from peers
I20260620 14:49:32.621526 11775 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 68faae2f3b5c445c898441cb1466402e [term 1 FOLLOWER]: Leader election won for term 1
I20260620 14:49:32.623879 11770 sys_catalog.cc:564] T 00000000000000000000000000000000 P 68faae2f3b5c445c898441cb1466402e [sys.catalog]: configured and running, proceeding with master startup.
I20260620 14:49:32.623965 11775 raft_consensus.cc:686] T 00000000000000000000000000000000 P 68faae2f3b5c445c898441cb1466402e [term 1 LEADER]: Becoming Leader. State: Replica: 68faae2f3b5c445c898441cb1466402e, State: Running, Role: LEADER
I20260620 14:49:32.624526 11775 consensus_queue.cc:237] T 00000000000000000000000000000000 P 68faae2f3b5c445c898441cb1466402e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "68faae2f3b5c445c898441cb1466402e" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 37829 } }
I20260620 14:49:32.629593 11777 sys_catalog.cc:455] T 00000000000000000000000000000000 P 68faae2f3b5c445c898441cb1466402e [sys.catalog]: SysCatalogTable state changed. Reason: New leader 68faae2f3b5c445c898441cb1466402e. Latest consensus state: current_term: 1 leader_uuid: "68faae2f3b5c445c898441cb1466402e" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "68faae2f3b5c445c898441cb1466402e" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 37829 } } }
I20260620 14:49:32.630100 11777 sys_catalog.cc:458] T 00000000000000000000000000000000 P 68faae2f3b5c445c898441cb1466402e [sys.catalog]: This master's current role is: LEADER
I20260620 14:49:32.632730 11776 sys_catalog.cc:455] T 00000000000000000000000000000000 P 68faae2f3b5c445c898441cb1466402e [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "68faae2f3b5c445c898441cb1466402e" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "68faae2f3b5c445c898441cb1466402e" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 37829 } } }
I20260620 14:49:32.632997 11787 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 14:49:32.633023 11776 sys_catalog.cc:458] T 00000000000000000000000000000000 P 68faae2f3b5c445c898441cb1466402e [sys.catalog]: This master's current role is: LEADER
I20260620 14:49:32.633307 11770 ranger_client.cc:318] Using new properties file: /tmp/dist-test-taskUci3DU/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781966829091300-388-0/minicluster-data/master-0/logs/kudu-ranger-subprocess-log4j2.properties
I20260620 14:49:32.646014 11787 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 14:49:32.663007 11787 catalog_manager.cc:1347] Generated new cluster ID: 699250d6e8cc492698d588286a3c53b9
I20260620 14:49:32.663138 11787 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 14:49:32.674096 11787 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 14:49:32.676501 11787 catalog_manager.cc:1504] Loading token signing keys...
I20260620 14:49:32.694099 11787 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 68faae2f3b5c445c898441cb1466402e: Generated new TSK 0
I20260620 14:49:32.695092 11787 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.0.97.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskUci3DU/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781966829091300-388-0/krb5kdc/kudu_127.0.97.1.keytab.
Entry for principal kudu/127.0.97.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskUci3DU/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781966829091300-388-0/krb5kdc/kudu_127.0.97.1.keytab.
I20260620 14:49:33.479190 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskUci3DU/build/asan/bin/kudu
/tmp/dist-test-taskUci3DU/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskUci3DU/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781966829091300-388-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskUci3DU/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781966829091300-388-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskUci3DU/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781966829091300-388-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskUci3DU/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781966829091300-388-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.0.97.20:52961/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.1:0
--local_ip_for_outbound_sockets=127.0.97.1
--webserver_interface=127.0.97.1
--webserver_port=0
--tserver_master_addrs=127.0.97.62:37829
--keytab_file=/tmp/dist-test-taskUci3DU/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781966829091300-388-0/krb5kdc/kudu_127.0.97.1.keytab
--principal=kudu/127.0.97.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:41959
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskUci3DU/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781966829091300-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskUci3DU/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781966829091300-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskUci3DU/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781966829091300-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Jun 20 14:49:33 dist-test-slave-q9j6 krb5kdc[11259](info): AS_REQ (1 etypes {17}) 127.0.0.1: ISSUE: authtime 1781966973, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
W20260620 14:49:33.940864 11815 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 14:49:33.941277 11815 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 14:49:33.941493 11815 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 14:49:33.954043 11815 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 14:49:33.954272 11815 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 14:49:33.954500 11815 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.1
I20260620 14:49:33.971411 11815 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:41959
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskUci3DU/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781966829091300-388-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskUci3DU/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781966829091300-388-0/minicluster-data/ts-0/wal
--ranger_kms_url=127.0.97.20:52961/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskUci3DU/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781966829091300-388-0/krb5kdc/kudu_127.0.97.1.keytab
--principal=kudu/127.0.97.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskUci3DU/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781966829091300-388-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.97.1
--webserver_port=0
--tserver_master_addrs=127.0.97.62:37829
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.1
--log_dir=/tmp/dist-test-taskUci3DU/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781966829091300-388-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 13:41:20 UTC on ccycloud.quasar-vdagoi.root.comops.site
build id 4457462
ASAN enabled
I20260620 14:49:33.973155 11815 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 14:49:33.974807 11815 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 14:49:33.993217 11826 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 14:49:33.997932 11815 server_base.cc:1034] running on GCE node
W20260620 14:49:33.998099 11825 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 14:49:33.998615 11828 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 14:49:33.999533 11815 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 14:49:34.003645 11815 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 14:49:34.004973 11815 hybrid_clock.cc:648] HybridClock initialized: now 1781966974004841 us; error 119 us; skew 500 ppm
I20260620 14:49:34.005400 11815 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 14:49:34 dist-test-slave-q9j6 krb5kdc[11259](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781966974, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 14:49:34.010176 11815 init.cc:377] Logged in from keytab as kudu/127.0.97.1@KRBTEST.COM (short username kudu)
I20260620 14:49:34.013679 11815 webserver.cc:460] Webserver started at http://127.0.97.1:45669/ using document root <none> and password file <none>
I20260620 14:49:34.014705 11815 fs_manager.cc:359] Metadata directory not provided
I20260620 14:49:34.014875 11815 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 14:49:34.015300 11815 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Jun 20 14:49:34 dist-test-slave-q9j6 krb5kdc[11259](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781966974, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.1@KRBTEST.COM for HTTP/127.0.97.20@KRBTEST.COM
I20260620 14:49:34.046756 11815 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskUci3DU/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781966829091300-388-0/minicluster-data/ts-0/data/instance:
uuid: "5e9007ef0cd447bbae5844cef38ba52c"
format_stamp: "Formatted at 2026-06-20 14:49:34 on dist-test-slave-q9j6"
server_key: "c564b0bd1b25a3ff5ae39c7192136ebe"
server_key_iv: "e02486ab19d36d4066557420f3095e9d"
server_key_version: "kuduclusterkey@0"
I20260620 14:49:34.048815 11815 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskUci3DU/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781966829091300-388-0/minicluster-data/ts-0/wal/instance:
uuid: "5e9007ef0cd447bbae5844cef38ba52c"
format_stamp: "Formatted at 2026-06-20 14:49:34 on dist-test-slave-q9j6"
server_key: "c564b0bd1b25a3ff5ae39c7192136ebe"
server_key_iv: "e02486ab19d36d4066557420f3095e9d"
server_key_version: "kuduclusterkey@0"
I20260620 14:49:34.056442 11815 fs_manager.cc:691] Time spent creating directory manager: real 0.007s user 0.006s sys 0.001s
I20260620 14:49:34.084414 11835 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 14:49:34.087109 11815 fs_manager.cc:725] Time spent opening block manager: real 0.012s user 0.011s sys 0.000s
I20260620 14:49:34.087393 11815 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskUci3DU/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781966829091300-388-0/minicluster-data/ts-0/data,/tmp/dist-test-taskUci3DU/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781966829091300-388-0/minicluster-data/ts-0/wal
uuid: "5e9007ef0cd447bbae5844cef38ba52c"
format_stamp: "Formatted at 2026-06-20 14:49:34 on dist-test-slave-q9j6"
server_key: "c564b0bd1b25a3ff5ae39c7192136ebe"
server_key_iv: "e02486ab19d36d4066557420f3095e9d"
server_key_version: "kuduclusterkey@0"
I20260620 14:49:34.087815 11815 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskUci3DU/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781966829091300-388-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskUci3DU/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781966829091300-388-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskUci3DU/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781966829091300-388-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 14:49:34.123690 11815 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 14:49:34.129403 11815 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 14:49:34.129801 11815 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 14:49:34.132032 11815 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 14:49:34.135214 11815 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 14:49:34.135325 11815 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 14:49:34.135509 11815 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 14:49:34.135578 11815 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 14:49:34.221570 11815 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.1:43109
I20260620 14:49:34.221588 11947 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.1:43109 every 8 connection(s)
I20260620 14:49:34.223357 11815 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskUci3DU/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781966829091300-388-0/minicluster-data/ts-0/data/info.pb
I20260620 14:49:34.231259 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskUci3DU/build/asan/bin/kudu as pid 11815
I20260620 14:49:34.231499 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskUci3DU/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781966829091300-388-0/minicluster-data/ts-0/wal/instance
I20260620 14:49:34.232407 388 external_mini_cluster.cc:1428] Setting key ef4e9a97310f89d570c9b65bb8394494
Jun 20 14:49:34 dist-test-slave-q9j6 krb5kdc[11259](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781966974, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.1@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260620 14:49:34.267002 11951 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:49:34.227899 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.1:34003 (local address 127.0.97.62:37829)
0620 14:49:34.245831 (+ 17932us) server_negotiation.cc:206] Beginning negotiation
0620 14:49:34.245838 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0620 14:49:34.245860 (+ 22us) server_negotiation.cc:407] Connection header received
0620 14:49:34.245927 (+ 67us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:49:34.245932 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:49:34.246023 (+ 91us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:49:34.246148 (+ 125us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:49:34.248397 (+ 2249us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:49:34.249681 (+ 1284us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:49:34.252577 (+ 2896us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:49:34.253391 (+ 814us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:49:34.253506 (+ 115us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:49:34.258435 (+ 4929us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 14:49:34.258443 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 14:49:34.258448 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 14:49:34.258493 (+ 45us) server_negotiation.cc:922] calling sasl_server_start()
0620 14:49:34.262187 (+ 3694us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:49:34.263380 (+ 1193us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:49:34.263386 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:49:34.263389 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:49:34.263585 (+ 196us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:49:34.264263 (+ 678us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:49:34.264268 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:49:34.264271 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:49:34.265111 (+ 840us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 14:49:34.265254 (+ 143us) server_negotiation.cc:1035] Waiting for connection context
0620 14:49:34.266346 (+ 1092us) server_negotiation.cc:299] Negotiation successful
0620 14:49:34.266614 (+ 268us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":17768,"thread_start_us":17570,"threads_started":1}
WARNING: no policy specified for kudu/127.0.97.2@KRBTEST.COM; defaulting to no policy
I20260620 14:49:34.269760 11950 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:49:34.227248 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:37829 (local address 127.0.97.1:34003)
0620 14:49:34.240952 (+ 13704us) negotiation.cc:107] Waiting for socket to connect
0620 14:49:34.241057 (+ 105us) client_negotiation.cc:174] Beginning negotiation
0620 14:49:34.244793 (+ 3736us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 14:49:34.246660 (+ 1867us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 14:49:34.246675 (+ 15us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 14:49:34.247428 (+ 753us) client_negotiation.cc:189] Negotiated authn=SASL
0620 14:49:34.248178 (+ 750us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:49:34.248196 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:49:34.249892 (+ 1696us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:49:34.249901 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:49:34.252360 (+ 2459us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:49:34.252390 (+ 30us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:49:34.253882 (+ 1492us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:49:34.253891 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:49:34.254240 (+ 349us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:49:34.255104 (+ 864us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 14:49:34.255137 (+ 33us) client_negotiation.cc:656] calling sasl_client_start()
0620 14:49:34.258095 (+ 2958us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 14:49:34.262460 (+ 4365us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 14:49:34.262471 (+ 11us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 14:49:34.262494 (+ 23us) client_negotiation.cc:760] calling sasl_client_step()
0620 14:49:34.263204 (+ 710us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 14:49:34.263788 (+ 584us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 14:49:34.263794 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 14:49:34.263801 (+ 7us) client_negotiation.cc:760] calling sasl_client_step()
0620 14:49:34.264119 (+ 318us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 14:49:34.265368 (+ 1249us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 14:49:34.265377 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 14:49:34.265864 (+ 487us) client_negotiation.cc:769] Sending connection context
0620 14:49:34.267581 (+ 1717us) client_negotiation.cc:240] Negotiation successful
0620 14:49:34.268198 (+ 617us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":13232,"thread_start_us":12915,"threads_started":1}
I20260620 14:49:34.280529 11948 heartbeater.cc:347] Connected to a master server at 127.0.97.62:37829
I20260620 14:49:34.280957 11948 heartbeater.cc:464] Registering TS with master...
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.2@KRBTEST.COM" created.
I20260620 14:49:34.281940 11948 heartbeater.cc:511] Master 127.0.97.62:37829 requested a full tablet report, sending...
I20260620 14:49:34.285043 11735 ts_manager.cc:194] Registered new tserver with Master: 5e9007ef0cd447bbae5844cef38ba52c (127.0.97.1:43109)
I20260620 14:49:34.287014 11735 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.97.1@KRBTEST.COM'} at 127.0.97.1:34003
Jun 20 14:49:34 dist-test-slave-q9j6 krb5kdc[11259](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781966973, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.62@KRBTEST.COM for HTTP/127.0.97.20@KRBTEST.COM
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskUci3DU/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781966829091300-388-0/krb5kdc/kudu_127.0.97.2.keytab.
Entry for principal kudu/127.0.97.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskUci3DU/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781966829091300-388-0/krb5kdc/kudu_127.0.97.2.keytab.
I20260620 14:49:34.327265 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskUci3DU/build/asan/bin/kudu
/tmp/dist-test-taskUci3DU/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskUci3DU/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781966829091300-388-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskUci3DU/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781966829091300-388-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskUci3DU/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781966829091300-388-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskUci3DU/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781966829091300-388-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.0.97.20:52961/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.2:0
--local_ip_for_outbound_sockets=127.0.97.2
--webserver_interface=127.0.97.2
--webserver_port=0
--tserver_master_addrs=127.0.97.62:37829
--keytab_file=/tmp/dist-test-taskUci3DU/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781966829091300-388-0/krb5kdc/kudu_127.0.97.2.keytab
--principal=kudu/127.0.97.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:41959
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskUci3DU/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781966829091300-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskUci3DU/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781966829091300-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskUci3DU/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781966829091300-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Jun 20 14:49:34 dist-test-slave-q9j6 krb5kdc[11259](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781966973, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.62@KRBTEST.COM for HTTP/127.0.97.20@KRBTEST.COM
W20260620 14:49:34.829982 11954 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 14:49:34.830430 11954 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 14:49:34.830647 11954 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 14:49:34.842772 11954 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 14:49:34.843044 11954 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 14:49:34.843278 11954 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.2
I20260620 14:49:34.859728 11954 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:41959
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskUci3DU/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781966829091300-388-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskUci3DU/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781966829091300-388-0/minicluster-data/ts-1/wal
--ranger_kms_url=127.0.97.20:52961/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskUci3DU/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781966829091300-388-0/krb5kdc/kudu_127.0.97.2.keytab
--principal=kudu/127.0.97.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskUci3DU/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781966829091300-388-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.97.2
--webserver_port=0
--tserver_master_addrs=127.0.97.62:37829
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.2
--log_dir=/tmp/dist-test-taskUci3DU/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781966829091300-388-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 13:41:20 UTC on ccycloud.quasar-vdagoi.root.comops.site
build id 4457462
ASAN enabled
I20260620 14:49:34.861961 11954 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 14:49:34.863999 11954 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 14:49:34.882225 11962 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 14:49:34.882483 11961 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 14:49:34.882225 11964 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 14:49:34.883558 11954 server_base.cc:1034] running on GCE node
I20260620 14:49:34.884245 11954 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 14:49:34.885371 11954 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 14:49:34.886662 11954 hybrid_clock.cc:648] HybridClock initialized: now 1781966974886597 us; error 61 us; skew 500 ppm
I20260620 14:49:34.887009 11954 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 14:49:34 dist-test-slave-q9j6 krb5kdc[11259](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781966974, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 14:49:34.890431 11954 init.cc:377] Logged in from keytab as kudu/127.0.97.2@KRBTEST.COM (short username kudu)
I20260620 14:49:34.895669 11954 webserver.cc:460] Webserver started at http://127.0.97.2:42047/ using document root <none> and password file <none>
I20260620 14:49:34.896466 11954 fs_manager.cc:359] Metadata directory not provided
I20260620 14:49:34.896620 11954 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 14:49:34.896975 11954 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Jun 20 14:49:34 dist-test-slave-q9j6 krb5kdc[11259](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781966974, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.2@KRBTEST.COM for HTTP/127.0.97.20@KRBTEST.COM
I20260620 14:49:34.917058 11954 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskUci3DU/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781966829091300-388-0/minicluster-data/ts-1/data/instance:
uuid: "d89ebf08e0fa4217a42f41b28afd44bf"
format_stamp: "Formatted at 2026-06-20 14:49:34 on dist-test-slave-q9j6"
server_key: "24d32488352073fb3906b9634d2619c8"
server_key_iv: "7cc8d52ec974ce374fdf5449cd7296a8"
server_key_version: "kuduclusterkey@0"
I20260620 14:49:34.918054 11954 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskUci3DU/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781966829091300-388-0/minicluster-data/ts-1/wal/instance:
uuid: "d89ebf08e0fa4217a42f41b28afd44bf"
format_stamp: "Formatted at 2026-06-20 14:49:34 on dist-test-slave-q9j6"
server_key: "24d32488352073fb3906b9634d2619c8"
server_key_iv: "7cc8d52ec974ce374fdf5449cd7296a8"
server_key_version: "kuduclusterkey@0"
I20260620 14:49:34.927289 11954 fs_manager.cc:691] Time spent creating directory manager: real 0.008s user 0.007s sys 0.000s
I20260620 14:49:34.947547 11978 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 14:49:34.949653 11954 fs_manager.cc:725] Time spent opening block manager: real 0.005s user 0.004s sys 0.000s
I20260620 14:49:34.949872 11954 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskUci3DU/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781966829091300-388-0/minicluster-data/ts-1/data,/tmp/dist-test-taskUci3DU/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781966829091300-388-0/minicluster-data/ts-1/wal
uuid: "d89ebf08e0fa4217a42f41b28afd44bf"
format_stamp: "Formatted at 2026-06-20 14:49:34 on dist-test-slave-q9j6"
server_key: "24d32488352073fb3906b9634d2619c8"
server_key_iv: "7cc8d52ec974ce374fdf5449cd7296a8"
server_key_version: "kuduclusterkey@0"
I20260620 14:49:34.950142 11954 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskUci3DU/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781966829091300-388-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskUci3DU/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781966829091300-388-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskUci3DU/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781966829091300-388-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 14:49:34.975724 11954 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 14:49:34.980789 11954 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 14:49:34.981122 11954 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 14:49:34.982282 11954 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 14:49:34.984117 11954 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 14:49:34.984216 11954 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 14:49:34.984349 11954 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 14:49:34.984416 11954 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 14:49:35.021515 11954 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.2:32829
I20260620 14:49:35.021554 12090 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.2:32829 every 8 connection(s)
I20260620 14:49:35.022842 11954 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskUci3DU/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781966829091300-388-0/minicluster-data/ts-1/data/info.pb
I20260620 14:49:35.031713 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskUci3DU/build/asan/bin/kudu as pid 11954
I20260620 14:49:35.031880 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskUci3DU/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781966829091300-388-0/minicluster-data/ts-1/wal/instance
I20260620 14:49:35.032316 388 external_mini_cluster.cc:1428] Setting key 0ef90ea21f0a59d1132c9349670c33e2
Jun 20 14:49:35 dist-test-slave-q9j6 krb5kdc[11259](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781966974, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.2@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260620 14:49:35.044098 12093 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:49:35.025470 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.2:44367 (local address 127.0.97.62:37829)
0620 14:49:35.025922 (+ 452us) server_negotiation.cc:206] Beginning negotiation
0620 14:49:35.025927 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 14:49:35.026962 (+ 1035us) server_negotiation.cc:407] Connection header received
0620 14:49:35.029004 (+ 2042us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:49:35.029010 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:49:35.029106 (+ 96us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:49:35.029230 (+ 124us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:49:35.030728 (+ 1498us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:49:35.031499 (+ 771us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:49:35.033368 (+ 1869us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:49:35.033822 (+ 454us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:49:35.033964 (+ 142us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:49:35.037608 (+ 3644us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 14:49:35.037613 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 14:49:35.037616 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 14:49:35.037655 (+ 39us) server_negotiation.cc:922] calling sasl_server_start()
0620 14:49:35.040825 (+ 3170us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:49:35.041658 (+ 833us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:49:35.041665 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:49:35.041673 (+ 8us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:49:35.041798 (+ 125us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:49:35.042244 (+ 446us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:49:35.042253 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:49:35.042256 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:49:35.042535 (+ 279us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 14:49:35.042704 (+ 169us) server_negotiation.cc:1035] Waiting for connection context
0620 14:49:35.043412 (+ 708us) server_negotiation.cc:299] Negotiation successful
0620 14:49:35.043710 (+ 298us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":1,"server-negotiator.queue_time_us":340,"thread_start_us":231,"threads_started":1}
I20260620 14:49:35.045094 12094 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:49:35.026089 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:37829 (local address 127.0.97.2:44367)
0620 14:49:35.026787 (+ 698us) negotiation.cc:107] Waiting for socket to connect
0620 14:49:35.026836 (+ 49us) client_negotiation.cc:174] Beginning negotiation
0620 14:49:35.028713 (+ 1877us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 14:49:35.029481 (+ 768us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 14:49:35.029489 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 14:49:35.029935 (+ 446us) client_negotiation.cc:189] Negotiated authn=SASL
0620 14:49:35.030537 (+ 602us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:49:35.030555 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:49:35.031708 (+ 1153us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:49:35.031713 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:49:35.033207 (+ 1494us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:49:35.033220 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:49:35.034065 (+ 845us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:49:35.034072 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:49:35.034324 (+ 252us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:49:35.035053 (+ 729us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 14:49:35.035075 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0620 14:49:35.037370 (+ 2295us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 14:49:35.040999 (+ 3629us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 14:49:35.041009 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 14:49:35.041030 (+ 21us) client_negotiation.cc:760] calling sasl_client_step()
0620 14:49:35.041512 (+ 482us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 14:49:35.041918 (+ 406us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 14:49:35.041923 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 14:49:35.041927 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0620 14:49:35.042118 (+ 191us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 14:49:35.042672 (+ 554us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 14:49:35.042681 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 14:49:35.043027 (+ 346us) client_negotiation.cc:769] Sending connection context
0620 14:49:35.043320 (+ 293us) client_negotiation.cc:240] Negotiation successful
0620 14:49:35.043907 (+ 587us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":471,"thread_start_us":291,"threads_started":1}
I20260620 14:49:35.046630 12091 heartbeater.cc:347] Connected to a master server at 127.0.97.62:37829
I20260620 14:49:35.046958 12091 heartbeater.cc:464] Registering TS with master...
I20260620 14:49:35.047798 12091 heartbeater.cc:511] Master 127.0.97.62:37829 requested a full tablet report, sending...
I20260620 14:49:35.049417 11735 ts_manager.cc:194] Registered new tserver with Master: d89ebf08e0fa4217a42f41b28afd44bf (127.0.97.2:32829)
I20260620 14:49:35.050233 11735 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.97.2@KRBTEST.COM'} at 127.0.97.2:44367
WARNING: no policy specified for kudu/127.0.97.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskUci3DU/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781966829091300-388-0/krb5kdc/kudu_127.0.97.3.keytab.
Entry for principal kudu/127.0.97.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskUci3DU/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781966829091300-388-0/krb5kdc/kudu_127.0.97.3.keytab.
I20260620 14:49:35.083097 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskUci3DU/build/asan/bin/kudu
/tmp/dist-test-taskUci3DU/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskUci3DU/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781966829091300-388-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskUci3DU/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781966829091300-388-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskUci3DU/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781966829091300-388-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskUci3DU/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781966829091300-388-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.0.97.20:52961/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.3:0
--local_ip_for_outbound_sockets=127.0.97.3
--webserver_interface=127.0.97.3
--webserver_port=0
--tserver_master_addrs=127.0.97.62:37829
--keytab_file=/tmp/dist-test-taskUci3DU/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781966829091300-388-0/krb5kdc/kudu_127.0.97.3.keytab
--principal=kudu/127.0.97.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:41959
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskUci3DU/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781966829091300-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskUci3DU/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781966829091300-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskUci3DU/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781966829091300-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260620 14:49:35.290952 11948 heartbeater.cc:503] Master 127.0.97.62:37829 was elected leader, sending a full tablet report...
W20260620 14:49:35.438103 12097 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 14:49:35.438403 12097 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 14:49:35.438498 12097 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 14:49:35.445786 12097 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 14:49:35.445902 12097 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 14:49:35.446014 12097 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.3
I20260620 14:49:35.456337 12097 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:41959
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskUci3DU/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781966829091300-388-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskUci3DU/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781966829091300-388-0/minicluster-data/ts-2/wal
--ranger_kms_url=127.0.97.20:52961/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskUci3DU/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781966829091300-388-0/krb5kdc/kudu_127.0.97.3.keytab
--principal=kudu/127.0.97.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskUci3DU/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781966829091300-388-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.97.3
--webserver_port=0
--tserver_master_addrs=127.0.97.62:37829
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.3
--log_dir=/tmp/dist-test-taskUci3DU/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781966829091300-388-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 13:41:20 UTC on ccycloud.quasar-vdagoi.root.comops.site
build id 4457462
ASAN enabled
I20260620 14:49:35.457747 12097 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 14:49:35.459100 12097 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 14:49:35.473266 12102 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 14:49:35.473361 12105 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 14:49:35.473289 12103 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 14:49:35.474232 12097 server_base.cc:1034] running on GCE node
I20260620 14:49:35.474874 12097 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 14:49:35.476003 12097 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 14:49:35.477217 12097 hybrid_clock.cc:648] HybridClock initialized: now 1781966975477146 us; error 59 us; skew 500 ppm
I20260620 14:49:35.477567 12097 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 14:49:35 dist-test-slave-q9j6 krb5kdc[11259](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781966975, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 14:49:35.481407 12097 init.cc:377] Logged in from keytab as kudu/127.0.97.3@KRBTEST.COM (short username kudu)
I20260620 14:49:35.482993 12097 webserver.cc:460] Webserver started at http://127.0.97.3:37781/ using document root <none> and password file <none>
I20260620 14:49:35.483770 12097 fs_manager.cc:359] Metadata directory not provided
I20260620 14:49:35.483879 12097 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 14:49:35.484148 12097 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Jun 20 14:49:35 dist-test-slave-q9j6 krb5kdc[11259](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781966975, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.3@KRBTEST.COM for HTTP/127.0.97.20@KRBTEST.COM
I20260620 14:49:35.500098 12097 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskUci3DU/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781966829091300-388-0/minicluster-data/ts-2/data/instance:
uuid: "5fa5e0a21f13421cad5b50a79d3a52b2"
format_stamp: "Formatted at 2026-06-20 14:49:35 on dist-test-slave-q9j6"
server_key: "669798ccece92f45ffb1153098bac004"
server_key_iv: "3b47536379dee43f2916c9dfbdd57c21"
server_key_version: "kuduclusterkey@0"
I20260620 14:49:35.501019 12097 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskUci3DU/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781966829091300-388-0/minicluster-data/ts-2/wal/instance:
uuid: "5fa5e0a21f13421cad5b50a79d3a52b2"
format_stamp: "Formatted at 2026-06-20 14:49:35 on dist-test-slave-q9j6"
server_key: "669798ccece92f45ffb1153098bac004"
server_key_iv: "3b47536379dee43f2916c9dfbdd57c21"
server_key_version: "kuduclusterkey@0"
I20260620 14:49:35.506628 12097 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.000s
I20260620 14:49:35.522325 12112 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 14:49:35.524070 12097 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.001s
I20260620 14:49:35.524286 12097 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskUci3DU/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781966829091300-388-0/minicluster-data/ts-2/data,/tmp/dist-test-taskUci3DU/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781966829091300-388-0/minicluster-data/ts-2/wal
uuid: "5fa5e0a21f13421cad5b50a79d3a52b2"
format_stamp: "Formatted at 2026-06-20 14:49:35 on dist-test-slave-q9j6"
server_key: "669798ccece92f45ffb1153098bac004"
server_key_iv: "3b47536379dee43f2916c9dfbdd57c21"
server_key_version: "kuduclusterkey@0"
I20260620 14:49:35.524537 12097 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskUci3DU/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781966829091300-388-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskUci3DU/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781966829091300-388-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskUci3DU/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781966829091300-388-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 14:49:35.548017 12097 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 14:49:35.552251 12097 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 14:49:35.552560 12097 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 14:49:35.553529 12097 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 14:49:35.554877 12097 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 14:49:35.554983 12097 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 14:49:35.555130 12097 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 14:49:35.555202 12097 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 14:49:35.591898 12097 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.3:36407
I20260620 14:49:35.591929 12224 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.3:36407 every 8 connection(s)
I20260620 14:49:35.593359 12097 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskUci3DU/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781966829091300-388-0/minicluster-data/ts-2/data/info.pb
I20260620 14:49:35.596875 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskUci3DU/build/asan/bin/kudu as pid 12097
I20260620 14:49:35.597086 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskUci3DU/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781966829091300-388-0/minicluster-data/ts-2/wal/instance
I20260620 14:49:35.597612 388 external_mini_cluster.cc:1428] Setting key 4cbdb2e6c6c3056fd59b3f1ab290ea2e
Jun 20 14:49:35 dist-test-slave-q9j6 krb5kdc[11259](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781966975, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.3@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260620 14:49:35.613092 12227 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:49:35.595991 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.3:60755 (local address 127.0.97.62:37829)
0620 14:49:35.596442 (+ 451us) server_negotiation.cc:206] Beginning negotiation
0620 14:49:35.596446 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 14:49:35.597658 (+ 1212us) server_negotiation.cc:407] Connection header received
0620 14:49:35.599391 (+ 1733us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:49:35.599396 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:49:35.599480 (+ 84us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:49:35.599608 (+ 128us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:49:35.600998 (+ 1390us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:49:35.601710 (+ 712us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:49:35.603234 (+ 1524us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:49:35.603752 (+ 518us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:49:35.603890 (+ 138us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:49:35.607025 (+ 3135us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 14:49:35.607030 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 14:49:35.607034 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 14:49:35.607065 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0620 14:49:35.609771 (+ 2706us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:49:35.610479 (+ 708us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:49:35.610482 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:49:35.610485 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:49:35.610615 (+ 130us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:49:35.611125 (+ 510us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:49:35.611130 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:49:35.611134 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:49:35.611469 (+ 335us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 14:49:35.612091 (+ 622us) server_negotiation.cc:1035] Waiting for connection context
0620 14:49:35.612422 (+ 331us) server_negotiation.cc:299] Negotiation successful
0620 14:49:35.612683 (+ 261us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":343,"thread_start_us":199,"threads_started":1}
I20260620 14:49:35.614142 12228 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:49:35.596629 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:37829 (local address 127.0.97.3:60755)
0620 14:49:35.597492 (+ 863us) negotiation.cc:107] Waiting for socket to connect
0620 14:49:35.597532 (+ 40us) client_negotiation.cc:174] Beginning negotiation
0620 14:49:35.599127 (+ 1595us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 14:49:35.599867 (+ 740us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 14:49:35.599875 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 14:49:35.600300 (+ 425us) client_negotiation.cc:189] Negotiated authn=SASL
0620 14:49:35.600837 (+ 537us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:49:35.600853 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:49:35.601862 (+ 1009us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:49:35.601873 (+ 11us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:49:35.603073 (+ 1200us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:49:35.603085 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:49:35.603978 (+ 893us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:49:35.603985 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:49:35.604226 (+ 241us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:49:35.604833 (+ 607us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 14:49:35.604859 (+ 26us) client_negotiation.cc:656] calling sasl_client_start()
0620 14:49:35.606824 (+ 1965us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 14:49:35.609921 (+ 3097us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 14:49:35.609928 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 14:49:35.609941 (+ 13us) client_negotiation.cc:760] calling sasl_client_step()
0620 14:49:35.610366 (+ 425us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 14:49:35.610731 (+ 365us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 14:49:35.610735 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 14:49:35.610738 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0620 14:49:35.610970 (+ 232us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 14:49:35.611624 (+ 654us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 14:49:35.611633 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 14:49:35.612002 (+ 369us) client_negotiation.cc:769] Sending connection context
0620 14:49:35.612323 (+ 321us) client_negotiation.cc:240] Negotiation successful
0620 14:49:35.612843 (+ 520us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":632,"thread_start_us":368,"threads_started":1}
I20260620 14:49:35.615460 12225 heartbeater.cc:347] Connected to a master server at 127.0.97.62:37829
I20260620 14:49:35.615814 12225 heartbeater.cc:464] Registering TS with master...
I20260620 14:49:35.616530 12225 heartbeater.cc:511] Master 127.0.97.62:37829 requested a full tablet report, sending...
I20260620 14:49:35.618006 11735 ts_manager.cc:194] Registered new tserver with Master: 5fa5e0a21f13421cad5b50a79d3a52b2 (127.0.97.3:36407)
I20260620 14:49:35.618813 11735 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.97.3@KRBTEST.COM'} at 127.0.97.3:60755
I20260620 14:49:35.625990 388 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260620 14:49:35.643782 12227 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:49:35.630646 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38276 (local address 127.0.97.62:37829)
0620 14:49:35.630882 (+ 236us) server_negotiation.cc:206] Beginning negotiation
0620 14:49:35.630890 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0620 14:49:35.631261 (+ 371us) server_negotiation.cc:407] Connection header received
0620 14:49:35.631463 (+ 202us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:49:35.631469 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:49:35.631559 (+ 90us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:49:35.631727 (+ 168us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:49:35.633095 (+ 1368us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:49:35.633863 (+ 768us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:49:35.635298 (+ 1435us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:49:35.635878 (+ 580us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:49:35.635977 (+ 99us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:49:35.638438 (+ 2461us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 14:49:35.638443 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 14:49:35.638447 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 14:49:35.638478 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0620 14:49:35.641341 (+ 2863us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:49:35.642009 (+ 668us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:49:35.642013 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:49:35.642015 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:49:35.642127 (+ 112us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:49:35.642516 (+ 389us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:49:35.642519 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:49:35.642521 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:49:35.642773 (+ 252us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 14:49:35.642937 (+ 164us) server_negotiation.cc:1035] Waiting for connection context
0620 14:49:35.643244 (+ 307us) server_negotiation.cc:299] Negotiation successful
0620 14:49:35.643413 (+ 169us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":67}
I20260620 14:49:35.648016 11735 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:38276:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260620 14:49:35.655023 11735 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260620 14:49:35.692158 12241 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:49:35.669650 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37764 (local address 127.0.97.2:32829)
0620 14:49:35.683031 (+ 13381us) server_negotiation.cc:206] Beginning negotiation
0620 14:49:35.683044 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0620 14:49:35.683078 (+ 34us) server_negotiation.cc:407] Connection header received
0620 14:49:35.683288 (+ 210us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:49:35.683303 (+ 15us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:49:35.683733 (+ 430us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:49:35.683993 (+ 260us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 14:49:35.685835 (+ 1842us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:49:35.687463 (+ 1628us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:49:35.689885 (+ 2422us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:49:35.690844 (+ 959us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:49:35.691019 (+ 175us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:49:35.691252 (+ 233us) server_negotiation.cc:1035] Waiting for connection context
0620 14:49:35.691624 (+ 372us) server_negotiation.cc:299] Negotiation successful
0620 14:49:35.691806 (+ 182us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":13194,"thread_start_us":13080,"threads_started":1}
I20260620 14:49:35.692937 12239 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:49:35.669304 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.2:32829 (local address 127.0.0.1:37764)
0620 14:49:35.682222 (+ 12918us) negotiation.cc:107] Waiting for socket to connect
0620 14:49:35.682250 (+ 28us) client_negotiation.cc:174] Beginning negotiation
0620 14:49:35.682434 (+ 184us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 14:49:35.684385 (+ 1951us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 14:49:35.684398 (+ 13us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 14:49:35.684442 (+ 44us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 14:49:35.685045 (+ 603us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:49:35.685058 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:49:35.687687 (+ 2629us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:49:35.687691 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:49:35.689718 (+ 2027us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:49:35.689734 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:49:35.691085 (+ 1351us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:49:35.691091 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:49:35.691360 (+ 269us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:49:35.691408 (+ 48us) client_negotiation.cc:769] Sending connection context
0620 14:49:35.691554 (+ 146us) client_negotiation.cc:240] Negotiation successful
0620 14:49:35.691799 (+ 245us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":12713,"mutex_wait_us":36,"spinlock_wait_cycles":8064,"thread_start_us":12488,"threads_started":1}
I20260620 14:49:35.695367 12243 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:49:35.683806 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.3:36407 (local address 127.0.0.1:34504)
0620 14:49:35.685884 (+ 2078us) negotiation.cc:107] Waiting for socket to connect
0620 14:49:35.685901 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0620 14:49:35.686014 (+ 113us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 14:49:35.686654 (+ 640us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 14:49:35.686659 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 14:49:35.686685 (+ 26us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 14:49:35.687461 (+ 776us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:49:35.687471 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:49:35.689030 (+ 1559us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:49:35.689038 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:49:35.691835 (+ 2797us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:49:35.691854 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:49:35.694324 (+ 2470us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:49:35.694331 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:49:35.694623 (+ 292us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:49:35.694749 (+ 126us) client_negotiation.cc:769] Sending connection context
0620 14:49:35.694848 (+ 99us) client_negotiation.cc:240] Negotiation successful
0620 14:49:35.694996 (+ 148us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1971,"mutex_wait_us":235,"thread_start_us":1784,"threads_started":1}
I20260620 14:49:35.695443 12240 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:49:35.669318 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44000 (local address 127.0.97.1:43109)
0620 14:49:35.683558 (+ 14240us) server_negotiation.cc:206] Beginning negotiation
0620 14:49:35.683565 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0620 14:49:35.683591 (+ 26us) server_negotiation.cc:407] Connection header received
0620 14:49:35.683743 (+ 152us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:49:35.683750 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:49:35.684021 (+ 271us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:49:35.684272 (+ 251us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 14:49:35.685773 (+ 1501us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:49:35.687463 (+ 1690us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:49:35.692088 (+ 4625us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:49:35.693756 (+ 1668us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:49:35.693868 (+ 112us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:49:35.694109 (+ 241us) server_negotiation.cc:1035] Waiting for connection context
0620 14:49:35.694932 (+ 823us) server_negotiation.cc:299] Negotiation successful
0620 14:49:35.695110 (+ 178us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":14048,"thread_start_us":13886,"threads_started":1}
I20260620 14:49:35.695446 12242 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:49:35.682839 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34504 (local address 127.0.97.3:36407)
0620 14:49:35.685334 (+ 2495us) server_negotiation.cc:206] Beginning negotiation
0620 14:49:35.685347 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0620 14:49:35.686132 (+ 785us) server_negotiation.cc:407] Connection header received
0620 14:49:35.686249 (+ 117us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:49:35.686257 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:49:35.686458 (+ 201us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:49:35.686675 (+ 217us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 14:49:35.687687 (+ 1012us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:49:35.688879 (+ 1192us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:49:35.692053 (+ 3174us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:49:35.693756 (+ 1703us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:49:35.693880 (+ 124us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:49:35.694158 (+ 278us) server_negotiation.cc:1035] Waiting for connection context
0620 14:49:35.694927 (+ 769us) server_negotiation.cc:299] Negotiation successful
0620 14:49:35.695094 (+ 167us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":2277,"thread_start_us":2085,"threads_started":1}
I20260620 14:49:35.696265 12238 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:49:35.669050 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:43109 (local address 127.0.0.1:44000)
0620 14:49:35.680860 (+ 11810us) negotiation.cc:107] Waiting for socket to connect
0620 14:49:35.680915 (+ 55us) client_negotiation.cc:174] Beginning negotiation
0620 14:49:35.681139 (+ 224us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 14:49:35.684609 (+ 3470us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 14:49:35.684621 (+ 12us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 14:49:35.684660 (+ 39us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 14:49:35.685569 (+ 909us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:49:35.685578 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:49:35.689099 (+ 3521us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:49:35.689103 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:49:35.691910 (+ 2807us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:49:35.691924 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:49:35.694336 (+ 2412us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:49:35.694341 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:49:35.694625 (+ 284us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:49:35.694753 (+ 128us) client_negotiation.cc:769] Sending connection context
0620 14:49:35.695726 (+ 973us) client_negotiation.cc:240] Negotiation successful
0620 14:49:35.695867 (+ 141us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":11513,"mutex_wait_us":871,"thread_start_us":11351,"threads_started":1}
I20260620 14:49:35.697590 12026 tablet_service.cc:1467] Processing CreateTablet for tablet 14896b5ab8194863ab12f17d9075944b (DEFAULT_TABLE table=test-table [id=1cb553e100104d97a319c83585c560ed]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 14:49:35.699165 12026 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 14896b5ab8194863ab12f17d9075944b. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 14:49:35.699787 12160 tablet_service.cc:1467] Processing CreateTablet for tablet 14896b5ab8194863ab12f17d9075944b (DEFAULT_TABLE table=test-table [id=1cb553e100104d97a319c83585c560ed]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 14:49:35.700120 11883 tablet_service.cc:1467] Processing CreateTablet for tablet 14896b5ab8194863ab12f17d9075944b (DEFAULT_TABLE table=test-table [id=1cb553e100104d97a319c83585c560ed]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 14:49:35.701287 12160 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 14896b5ab8194863ab12f17d9075944b. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 14:49:35.701399 11883 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 14896b5ab8194863ab12f17d9075944b. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 14:49:35.712263 12244 tablet_bootstrap.cc:492] T 14896b5ab8194863ab12f17d9075944b P 5e9007ef0cd447bbae5844cef38ba52c: Bootstrap starting.
I20260620 14:49:35.716310 12244 tablet_bootstrap.cc:654] T 14896b5ab8194863ab12f17d9075944b P 5e9007ef0cd447bbae5844cef38ba52c: Neither blocks nor log segments found. Creating new log.
I20260620 14:49:35.717940 12244 log.cc:824] T 14896b5ab8194863ab12f17d9075944b P 5e9007ef0cd447bbae5844cef38ba52c: Log is configured to *not* fsync() on all Append() calls
I20260620 14:49:35.719362 12245 tablet_bootstrap.cc:492] T 14896b5ab8194863ab12f17d9075944b P d89ebf08e0fa4217a42f41b28afd44bf: Bootstrap starting.
I20260620 14:49:35.721272 12244 tablet_bootstrap.cc:492] T 14896b5ab8194863ab12f17d9075944b P 5e9007ef0cd447bbae5844cef38ba52c: No bootstrap required, opened a new log
I20260620 14:49:35.721832 12244 ts_tablet_manager.cc:1397] T 14896b5ab8194863ab12f17d9075944b P 5e9007ef0cd447bbae5844cef38ba52c: Time spent bootstrapping tablet: real 0.010s user 0.008s sys 0.000s
I20260620 14:49:35.722119 12246 tablet_bootstrap.cc:492] T 14896b5ab8194863ab12f17d9075944b P 5fa5e0a21f13421cad5b50a79d3a52b2: Bootstrap starting.
I20260620 14:49:35.725512 12245 tablet_bootstrap.cc:654] T 14896b5ab8194863ab12f17d9075944b P d89ebf08e0fa4217a42f41b28afd44bf: Neither blocks nor log segments found. Creating new log.
I20260620 14:49:35.726550 12245 log.cc:824] T 14896b5ab8194863ab12f17d9075944b P d89ebf08e0fa4217a42f41b28afd44bf: Log is configured to *not* fsync() on all Append() calls
I20260620 14:49:35.729960 12246 tablet_bootstrap.cc:654] T 14896b5ab8194863ab12f17d9075944b P 5fa5e0a21f13421cad5b50a79d3a52b2: Neither blocks nor log segments found. Creating new log.
I20260620 14:49:35.730356 12245 tablet_bootstrap.cc:492] T 14896b5ab8194863ab12f17d9075944b P d89ebf08e0fa4217a42f41b28afd44bf: No bootstrap required, opened a new log
I20260620 14:49:35.730659 12245 ts_tablet_manager.cc:1397] T 14896b5ab8194863ab12f17d9075944b P d89ebf08e0fa4217a42f41b28afd44bf: Time spent bootstrapping tablet: real 0.012s user 0.004s sys 0.002s
I20260620 14:49:35.731191 12246 log.cc:824] T 14896b5ab8194863ab12f17d9075944b P 5fa5e0a21f13421cad5b50a79d3a52b2: Log is configured to *not* fsync() on all Append() calls
I20260620 14:49:35.731977 12244 raft_consensus.cc:348] T 14896b5ab8194863ab12f17d9075944b P 5e9007ef0cd447bbae5844cef38ba52c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5e9007ef0cd447bbae5844cef38ba52c" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 43109 } } peers { permanent_uuid: "d89ebf08e0fa4217a42f41b28afd44bf" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 32829 } } peers { permanent_uuid: "5fa5e0a21f13421cad5b50a79d3a52b2" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 36407 } }
I20260620 14:49:35.732578 12244 raft_consensus.cc:374] T 14896b5ab8194863ab12f17d9075944b P 5e9007ef0cd447bbae5844cef38ba52c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 14:49:35.732697 12244 raft_consensus.cc:729] T 14896b5ab8194863ab12f17d9075944b P 5e9007ef0cd447bbae5844cef38ba52c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5e9007ef0cd447bbae5844cef38ba52c, State: Initialized, Role: FOLLOWER
I20260620 14:49:35.733521 12244 consensus_queue.cc:260] T 14896b5ab8194863ab12f17d9075944b P 5e9007ef0cd447bbae5844cef38ba52c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5e9007ef0cd447bbae5844cef38ba52c" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 43109 } } peers { permanent_uuid: "d89ebf08e0fa4217a42f41b28afd44bf" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 32829 } } peers { permanent_uuid: "5fa5e0a21f13421cad5b50a79d3a52b2" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 36407 } }
I20260620 14:49:35.736388 12244 ts_tablet_manager.cc:1428] T 14896b5ab8194863ab12f17d9075944b P 5e9007ef0cd447bbae5844cef38ba52c: Time spent starting tablet: real 0.014s user 0.009s sys 0.004s
I20260620 14:49:35.741235 12245 raft_consensus.cc:348] T 14896b5ab8194863ab12f17d9075944b P d89ebf08e0fa4217a42f41b28afd44bf [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5e9007ef0cd447bbae5844cef38ba52c" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 43109 } } peers { permanent_uuid: "d89ebf08e0fa4217a42f41b28afd44bf" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 32829 } } peers { permanent_uuid: "5fa5e0a21f13421cad5b50a79d3a52b2" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 36407 } }
I20260620 14:49:35.741806 12245 raft_consensus.cc:374] T 14896b5ab8194863ab12f17d9075944b P d89ebf08e0fa4217a42f41b28afd44bf [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 14:49:35.741956 12245 raft_consensus.cc:729] T 14896b5ab8194863ab12f17d9075944b P d89ebf08e0fa4217a42f41b28afd44bf [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d89ebf08e0fa4217a42f41b28afd44bf, State: Initialized, Role: FOLLOWER
I20260620 14:49:35.742723 12246 tablet_bootstrap.cc:492] T 14896b5ab8194863ab12f17d9075944b P 5fa5e0a21f13421cad5b50a79d3a52b2: No bootstrap required, opened a new log
I20260620 14:49:35.742700 12245 consensus_queue.cc:260] T 14896b5ab8194863ab12f17d9075944b P d89ebf08e0fa4217a42f41b28afd44bf [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5e9007ef0cd447bbae5844cef38ba52c" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 43109 } } peers { permanent_uuid: "d89ebf08e0fa4217a42f41b28afd44bf" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 32829 } } peers { permanent_uuid: "5fa5e0a21f13421cad5b50a79d3a52b2" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 36407 } }
I20260620 14:49:35.743031 12246 ts_tablet_manager.cc:1397] T 14896b5ab8194863ab12f17d9075944b P 5fa5e0a21f13421cad5b50a79d3a52b2: Time spent bootstrapping tablet: real 0.021s user 0.011s sys 0.004s
I20260620 14:49:35.743840 12091 heartbeater.cc:503] Master 127.0.97.62:37829 was elected leader, sending a full tablet report...
I20260620 14:49:35.744760 12245 ts_tablet_manager.cc:1428] T 14896b5ab8194863ab12f17d9075944b P d89ebf08e0fa4217a42f41b28afd44bf: Time spent starting tablet: real 0.014s user 0.009s sys 0.004s
W20260620 14:49:35.750114 11949 tablet.cc:2369] T 14896b5ab8194863ab12f17d9075944b P 5e9007ef0cd447bbae5844cef38ba52c: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260620 14:49:35.750995 12251 raft_consensus.cc:482] T 14896b5ab8194863ab12f17d9075944b P d89ebf08e0fa4217a42f41b28afd44bf [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260620 14:49:35.751296 12251 raft_consensus.cc:504] T 14896b5ab8194863ab12f17d9075944b P d89ebf08e0fa4217a42f41b28afd44bf [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5e9007ef0cd447bbae5844cef38ba52c" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 43109 } } peers { permanent_uuid: "d89ebf08e0fa4217a42f41b28afd44bf" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 32829 } } peers { permanent_uuid: "5fa5e0a21f13421cad5b50a79d3a52b2" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 36407 } }
I20260620 14:49:35.752076 12246 raft_consensus.cc:348] T 14896b5ab8194863ab12f17d9075944b P 5fa5e0a21f13421cad5b50a79d3a52b2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5e9007ef0cd447bbae5844cef38ba52c" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 43109 } } peers { permanent_uuid: "d89ebf08e0fa4217a42f41b28afd44bf" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 32829 } } peers { permanent_uuid: "5fa5e0a21f13421cad5b50a79d3a52b2" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 36407 } }
I20260620 14:49:35.752563 12246 raft_consensus.cc:374] T 14896b5ab8194863ab12f17d9075944b P 5fa5e0a21f13421cad5b50a79d3a52b2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 14:49:35.752672 12246 raft_consensus.cc:729] T 14896b5ab8194863ab12f17d9075944b P 5fa5e0a21f13421cad5b50a79d3a52b2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5fa5e0a21f13421cad5b50a79d3a52b2, State: Initialized, Role: FOLLOWER
I20260620 14:49:35.753443 12251 leader_election.cc:288] T 14896b5ab8194863ab12f17d9075944b P d89ebf08e0fa4217a42f41b28afd44bf [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 5e9007ef0cd447bbae5844cef38ba52c (127.0.97.1:43109), 5fa5e0a21f13421cad5b50a79d3a52b2 (127.0.97.3:36407)
I20260620 14:49:35.753350 12246 consensus_queue.cc:260] T 14896b5ab8194863ab12f17d9075944b P 5fa5e0a21f13421cad5b50a79d3a52b2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5e9007ef0cd447bbae5844cef38ba52c" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 43109 } } peers { permanent_uuid: "d89ebf08e0fa4217a42f41b28afd44bf" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 32829 } } peers { permanent_uuid: "5fa5e0a21f13421cad5b50a79d3a52b2" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 36407 } }
I20260620 14:49:35.756978 12225 heartbeater.cc:503] Master 127.0.97.62:37829 was elected leader, sending a full tablet report...
I20260620 14:49:35.757896 12246 ts_tablet_manager.cc:1428] T 14896b5ab8194863ab12f17d9075944b P 5fa5e0a21f13421cad5b50a79d3a52b2: Time spent starting tablet: real 0.015s user 0.008s sys 0.008s
I20260620 14:49:35.762307 12242 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:49:35.753847 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.2:60841 (local address 127.0.97.3:36407)
0620 14:49:35.754092 (+ 245us) server_negotiation.cc:206] Beginning negotiation
0620 14:49:35.754102 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0620 14:49:35.755531 (+ 1429us) server_negotiation.cc:407] Connection header received
0620 14:49:35.755840 (+ 309us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:49:35.755845 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:49:35.755934 (+ 89us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:49:35.755998 (+ 64us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 14:49:35.756867 (+ 869us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:49:35.757906 (+ 1039us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:49:35.760416 (+ 2510us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:49:35.761291 (+ 875us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:49:35.761395 (+ 104us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:49:35.761470 (+ 75us) server_negotiation.cc:1035] Waiting for connection context
0620 14:49:35.761926 (+ 456us) server_negotiation.cc:299] Negotiation successful
0620 14:49:35.762055 (+ 129us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":72}
I20260620 14:49:35.762719 12254 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:49:35.753639 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.3:36407 (local address 127.0.97.2:60841)
0620 14:49:35.755425 (+ 1786us) negotiation.cc:107] Waiting for socket to connect
0620 14:49:35.755455 (+ 30us) client_negotiation.cc:174] Beginning negotiation
0620 14:49:35.755661 (+ 206us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 14:49:35.756260 (+ 599us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 14:49:35.756264 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 14:49:35.756287 (+ 23us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 14:49:35.756674 (+ 387us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:49:35.756686 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:49:35.758153 (+ 1467us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:49:35.758164 (+ 11us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:49:35.760264 (+ 2100us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:49:35.760278 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:49:35.761536 (+ 1258us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:49:35.761542 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:49:35.761723 (+ 181us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:49:35.761757 (+ 34us) client_negotiation.cc:769] Sending connection context
0620 14:49:35.761894 (+ 137us) client_negotiation.cc:240] Negotiation successful
0620 14:49:35.762048 (+ 154us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1670,"mutex_wait_us":723,"thread_start_us":409,"threads_started":1}
I20260620 14:49:35.762930 12240 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:49:35.753853 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.2:35647 (local address 127.0.97.1:43109)
0620 14:49:35.754269 (+ 416us) server_negotiation.cc:206] Beginning negotiation
0620 14:49:35.754284 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0620 14:49:35.754676 (+ 392us) server_negotiation.cc:407] Connection header received
0620 14:49:35.755072 (+ 396us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:49:35.755081 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:49:35.755183 (+ 102us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:49:35.755267 (+ 84us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 14:49:35.756290 (+ 1023us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:49:35.757465 (+ 1175us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:49:35.760310 (+ 2845us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:49:35.761717 (+ 1407us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:49:35.761843 (+ 126us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:49:35.761964 (+ 121us) server_negotiation.cc:1035] Waiting for connection context
0620 14:49:35.762534 (+ 570us) server_negotiation.cc:299] Negotiation successful
0620 14:49:35.762674 (+ 140us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":281}
I20260620 14:49:35.763733 12252 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:49:35.753639 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:43109 (local address 127.0.97.2:35647)
0620 14:49:35.754526 (+ 887us) negotiation.cc:107] Waiting for socket to connect
0620 14:49:35.754553 (+ 27us) client_negotiation.cc:174] Beginning negotiation
0620 14:49:35.754835 (+ 282us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 14:49:35.755531 (+ 696us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 14:49:35.755538 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 14:49:35.755583 (+ 45us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 14:49:35.756044 (+ 461us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:49:35.756060 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:49:35.757649 (+ 1589us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:49:35.757654 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:49:35.760096 (+ 2442us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:49:35.760113 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:49:35.762147 (+ 2034us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:49:35.762153 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:49:35.762342 (+ 189us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:49:35.762371 (+ 29us) client_negotiation.cc:769] Sending connection context
0620 14:49:35.763167 (+ 796us) client_negotiation.cc:240] Negotiation successful
0620 14:49:35.763281 (+ 114us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":676,"mutex_wait_us":31,"thread_start_us":468,"threads_started":1}
I20260620 14:49:35.763929 12180 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "14896b5ab8194863ab12f17d9075944b" candidate_uuid: "d89ebf08e0fa4217a42f41b28afd44bf" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5fa5e0a21f13421cad5b50a79d3a52b2" is_pre_election: true
I20260620 14:49:35.764515 12180 raft_consensus.cc:2454] T 14896b5ab8194863ab12f17d9075944b P 5fa5e0a21f13421cad5b50a79d3a52b2 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d89ebf08e0fa4217a42f41b28afd44bf in term 0.
I20260620 14:49:35.764825 11903 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "14896b5ab8194863ab12f17d9075944b" candidate_uuid: "d89ebf08e0fa4217a42f41b28afd44bf" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5e9007ef0cd447bbae5844cef38ba52c" is_pre_election: true
I20260620 14:49:35.765275 11979 leader_election.cc:302] T 14896b5ab8194863ab12f17d9075944b P d89ebf08e0fa4217a42f41b28afd44bf [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5fa5e0a21f13421cad5b50a79d3a52b2, d89ebf08e0fa4217a42f41b28afd44bf; no voters:
I20260620 14:49:35.765403 11903 raft_consensus.cc:2454] T 14896b5ab8194863ab12f17d9075944b P 5e9007ef0cd447bbae5844cef38ba52c [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d89ebf08e0fa4217a42f41b28afd44bf in term 0.
I20260620 14:49:35.765718 12251 raft_consensus.cc:2781] T 14896b5ab8194863ab12f17d9075944b P d89ebf08e0fa4217a42f41b28afd44bf [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260620 14:49:35.765913 12251 raft_consensus.cc:482] T 14896b5ab8194863ab12f17d9075944b P d89ebf08e0fa4217a42f41b28afd44bf [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260620 14:49:35.766048 12251 raft_consensus.cc:3037] T 14896b5ab8194863ab12f17d9075944b P d89ebf08e0fa4217a42f41b28afd44bf [term 0 FOLLOWER]: Advancing to term 1
I20260620 14:49:35.768486 12251 raft_consensus.cc:504] T 14896b5ab8194863ab12f17d9075944b P d89ebf08e0fa4217a42f41b28afd44bf [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5e9007ef0cd447bbae5844cef38ba52c" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 43109 } } peers { permanent_uuid: "d89ebf08e0fa4217a42f41b28afd44bf" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 32829 } } peers { permanent_uuid: "5fa5e0a21f13421cad5b50a79d3a52b2" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 36407 } }
I20260620 14:49:35.769248 12251 leader_election.cc:288] T 14896b5ab8194863ab12f17d9075944b P d89ebf08e0fa4217a42f41b28afd44bf [CANDIDATE]: Term 1 election: Requested vote from peers 5e9007ef0cd447bbae5844cef38ba52c (127.0.97.1:43109), 5fa5e0a21f13421cad5b50a79d3a52b2 (127.0.97.3:36407)
I20260620 14:49:35.769776 11903 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "14896b5ab8194863ab12f17d9075944b" candidate_uuid: "d89ebf08e0fa4217a42f41b28afd44bf" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5e9007ef0cd447bbae5844cef38ba52c"
I20260620 14:49:35.769793 12180 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "14896b5ab8194863ab12f17d9075944b" candidate_uuid: "d89ebf08e0fa4217a42f41b28afd44bf" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5fa5e0a21f13421cad5b50a79d3a52b2"
I20260620 14:49:35.769995 11903 raft_consensus.cc:3037] T 14896b5ab8194863ab12f17d9075944b P 5e9007ef0cd447bbae5844cef38ba52c [term 0 FOLLOWER]: Advancing to term 1
I20260620 14:49:35.770017 12180 raft_consensus.cc:3037] T 14896b5ab8194863ab12f17d9075944b P 5fa5e0a21f13421cad5b50a79d3a52b2 [term 0 FOLLOWER]: Advancing to term 1
I20260620 14:49:35.772758 11903 raft_consensus.cc:2454] T 14896b5ab8194863ab12f17d9075944b P 5e9007ef0cd447bbae5844cef38ba52c [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d89ebf08e0fa4217a42f41b28afd44bf in term 1.
I20260620 14:49:35.772867 12180 raft_consensus.cc:2454] T 14896b5ab8194863ab12f17d9075944b P 5fa5e0a21f13421cad5b50a79d3a52b2 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d89ebf08e0fa4217a42f41b28afd44bf in term 1.
I20260620 14:49:35.773350 11981 leader_election.cc:302] T 14896b5ab8194863ab12f17d9075944b P d89ebf08e0fa4217a42f41b28afd44bf [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5e9007ef0cd447bbae5844cef38ba52c, d89ebf08e0fa4217a42f41b28afd44bf; no voters:
I20260620 14:49:35.773725 12251 raft_consensus.cc:2781] T 14896b5ab8194863ab12f17d9075944b P d89ebf08e0fa4217a42f41b28afd44bf [term 1 FOLLOWER]: Leader election won for term 1
I20260620 14:49:35.774173 12251 raft_consensus.cc:686] T 14896b5ab8194863ab12f17d9075944b P d89ebf08e0fa4217a42f41b28afd44bf [term 1 LEADER]: Becoming Leader. State: Replica: d89ebf08e0fa4217a42f41b28afd44bf, State: Running, Role: LEADER
I20260620 14:49:35.774695 12251 consensus_queue.cc:237] T 14896b5ab8194863ab12f17d9075944b P d89ebf08e0fa4217a42f41b28afd44bf [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5e9007ef0cd447bbae5844cef38ba52c" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 43109 } } peers { permanent_uuid: "d89ebf08e0fa4217a42f41b28afd44bf" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 32829 } } peers { permanent_uuid: "5fa5e0a21f13421cad5b50a79d3a52b2" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 36407 } }
W20260620 14:49:35.774964 12092 tablet.cc:2369] T 14896b5ab8194863ab12f17d9075944b P d89ebf08e0fa4217a42f41b28afd44bf: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260620 14:49:35.780354 11735 catalog_manager.cc:5495] T 14896b5ab8194863ab12f17d9075944b P d89ebf08e0fa4217a42f41b28afd44bf reported cstate change: term changed from 0 to 1, leader changed from <none> to d89ebf08e0fa4217a42f41b28afd44bf (127.0.97.2). New cstate: current_term: 1 leader_uuid: "d89ebf08e0fa4217a42f41b28afd44bf" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5e9007ef0cd447bbae5844cef38ba52c" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 43109 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "d89ebf08e0fa4217a42f41b28afd44bf" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 32829 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "5fa5e0a21f13421cad5b50a79d3a52b2" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 36407 } health_report { overall_health: UNKNOWN } } }
I20260620 14:49:35.803983 12241 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:49:35.798630 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37770 (local address 127.0.97.2:32829)
0620 14:49:35.798843 (+ 213us) server_negotiation.cc:206] Beginning negotiation
0620 14:49:35.798848 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 14:49:35.798865 (+ 17us) server_negotiation.cc:407] Connection header received
0620 14:49:35.799048 (+ 183us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:49:35.799053 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:49:35.799141 (+ 88us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:49:35.799273 (+ 132us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 14:49:35.799647 (+ 374us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:49:35.800339 (+ 692us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:49:35.801739 (+ 1400us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:49:35.802117 (+ 378us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:49:35.802209 (+ 92us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:49:35.802710 (+ 501us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 14:49:35.803235 (+ 525us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0620 14:49:35.803347 (+ 112us) server_negotiation.cc:1035] Waiting for connection context
0620 14:49:35.803517 (+ 170us) server_negotiation.cc:299] Negotiation successful
0620 14:49:35.803587 (+ 70us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":76}
I20260620 14:49:35.812288 11903 raft_consensus.cc:1261] T 14896b5ab8194863ab12f17d9075944b P 5e9007ef0cd447bbae5844cef38ba52c [term 1 FOLLOWER]: Refusing update from remote peer d89ebf08e0fa4217a42f41b28afd44bf: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 14:49:35.812288 12180 raft_consensus.cc:1261] T 14896b5ab8194863ab12f17d9075944b P 5fa5e0a21f13421cad5b50a79d3a52b2 [term 1 FOLLOWER]: Refusing update from remote peer d89ebf08e0fa4217a42f41b28afd44bf: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 14:49:35.813342 12255 consensus_queue.cc:1035] T 14896b5ab8194863ab12f17d9075944b P d89ebf08e0fa4217a42f41b28afd44bf [LEADER]: Connected to new peer: Peer: permanent_uuid: "5e9007ef0cd447bbae5844cef38ba52c" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 43109 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 14:49:35.813726 12251 consensus_queue.cc:1035] T 14896b5ab8194863ab12f17d9075944b P d89ebf08e0fa4217a42f41b28afd44bf [LEADER]: Connected to new peer: Peer: permanent_uuid: "5fa5e0a21f13421cad5b50a79d3a52b2" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 36407 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 14:49:35.825891 12258 mvcc.cc:205] Tried to move back new op lower bound from 7298936732916731904 to 7298936732778348544. Current Snapshot: MvccSnapshot[applied={T|T < 7298936732916731904}]
I20260620 14:49:35.839972 11735 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:38276:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 14:49:35.840209 11735 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:38276:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 14:49:35.844173 11735 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 68faae2f3b5c445c898441cb1466402e: Sending DeleteTablet for 3 replicas of tablet 14896b5ab8194863ab12f17d9075944b
I20260620 14:49:35.845307 11883 tablet_service.cc:1514] Processing DeleteTablet for tablet 14896b5ab8194863ab12f17d9075944b with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 14:49:35 UTC) from {username='kudu', principal='kudu/127.0.97.62@KRBTEST.COM'} at 127.0.0.1:44000
I20260620 14:49:35.845719 12160 tablet_service.cc:1514] Processing DeleteTablet for tablet 14896b5ab8194863ab12f17d9075944b with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 14:49:35 UTC) from {username='kudu', principal='kudu/127.0.97.62@KRBTEST.COM'} at 127.0.0.1:34504
I20260620 14:49:35.846642 12269 tablet_replica.cc:330] T 14896b5ab8194863ab12f17d9075944b P 5fa5e0a21f13421cad5b50a79d3a52b2: stopping tablet replica
I20260620 14:49:35.847008 12026 tablet_service.cc:1514] Processing DeleteTablet for tablet 14896b5ab8194863ab12f17d9075944b with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 14:49:35 UTC) from {username='kudu', principal='kudu/127.0.97.62@KRBTEST.COM'} at 127.0.0.1:37764
I20260620 14:49:35.847126 12269 raft_consensus.cc:2229] T 14896b5ab8194863ab12f17d9075944b P 5fa5e0a21f13421cad5b50a79d3a52b2 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 14:49:35.847784 12269 raft_consensus.cc:2258] T 14896b5ab8194863ab12f17d9075944b P 5fa5e0a21f13421cad5b50a79d3a52b2 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 14:49:35.852098 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskUci3DU/build/asan/bin/kudu pid 11815
I20260620 14:49:35.853318 12269 ts_tablet_manager.cc:1905] T 14896b5ab8194863ab12f17d9075944b P 5fa5e0a21f13421cad5b50a79d3a52b2: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 14:49:35.858361 12270 tablet_replica.cc:330] T 14896b5ab8194863ab12f17d9075944b P d89ebf08e0fa4217a42f41b28afd44bf: stopping tablet replica
I20260620 14:49:35.858922 12270 raft_consensus.cc:2229] T 14896b5ab8194863ab12f17d9075944b P d89ebf08e0fa4217a42f41b28afd44bf [term 1 LEADER]: Raft consensus shutting down.
I20260620 14:49:35.859705 12270 raft_consensus.cc:2258] T 14896b5ab8194863ab12f17d9075944b P d89ebf08e0fa4217a42f41b28afd44bf [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 14:49:35.861070 12269 ts_tablet_manager.cc:1918] T 14896b5ab8194863ab12f17d9075944b P 5fa5e0a21f13421cad5b50a79d3a52b2: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260620 14:49:35.861295 12269 log.cc:1196] T 14896b5ab8194863ab12f17d9075944b P 5fa5e0a21f13421cad5b50a79d3a52b2: Deleting WAL directory at /tmp/dist-test-taskUci3DU/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781966829091300-388-0/minicluster-data/ts-2/wal/wals/14896b5ab8194863ab12f17d9075944b
I20260620 14:49:35.861868 12269 ts_tablet_manager.cc:1939] T 14896b5ab8194863ab12f17d9075944b P 5fa5e0a21f13421cad5b50a79d3a52b2: Deleting consensus metadata
I20260620 14:49:35.862133 12270 ts_tablet_manager.cc:1905] T 14896b5ab8194863ab12f17d9075944b P d89ebf08e0fa4217a42f41b28afd44bf: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 14:49:35.863932 11719 catalog_manager.cc:4841] TS 5fa5e0a21f13421cad5b50a79d3a52b2 (127.0.97.3:36407): tablet 14896b5ab8194863ab12f17d9075944b (table test-table [id=1cb553e100104d97a319c83585c560ed]) successfully deleted
I20260620 14:49:35.870061 12270 ts_tablet_manager.cc:1918] T 14896b5ab8194863ab12f17d9075944b P d89ebf08e0fa4217a42f41b28afd44bf: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260620 14:49:35.870275 12270 log.cc:1196] T 14896b5ab8194863ab12f17d9075944b P d89ebf08e0fa4217a42f41b28afd44bf: Deleting WAL directory at /tmp/dist-test-taskUci3DU/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781966829091300-388-0/minicluster-data/ts-1/wal/wals/14896b5ab8194863ab12f17d9075944b
I20260620 14:49:35.870807 12270 ts_tablet_manager.cc:1939] T 14896b5ab8194863ab12f17d9075944b P d89ebf08e0fa4217a42f41b28afd44bf: Deleting consensus metadata
I20260620 14:49:35.871096 12268 tablet_replica.cc:330] T 14896b5ab8194863ab12f17d9075944b P 5e9007ef0cd447bbae5844cef38ba52c: stopping tablet replica
I20260620 14:49:35.871588 12268 raft_consensus.cc:2229] T 14896b5ab8194863ab12f17d9075944b P 5e9007ef0cd447bbae5844cef38ba52c [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 14:49:35.872359 12268 raft_consensus.cc:2258] T 14896b5ab8194863ab12f17d9075944b P 5e9007ef0cd447bbae5844cef38ba52c [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 14:49:35.872649 11719 catalog_manager.cc:4841] TS d89ebf08e0fa4217a42f41b28afd44bf (127.0.97.2:32829): tablet 14896b5ab8194863ab12f17d9075944b (table test-table [id=1cb553e100104d97a319c83585c560ed]) successfully deleted
I20260620 14:49:35.874457 12268 ts_tablet_manager.cc:1905] T 14896b5ab8194863ab12f17d9075944b P 5e9007ef0cd447bbae5844cef38ba52c: Deleting tablet data with delete state TABLET_DATA_DELETED
Jun 20 14:49:35 dist-test-slave-q9j6 krb5kdc[11259](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781966971, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.1@KRBTEST.COM
I20260620 14:49:35.881059 12268 ts_tablet_manager.cc:1918] T 14896b5ab8194863ab12f17d9075944b P 5e9007ef0cd447bbae5844cef38ba52c: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260620 14:49:35.881335 12268 log.cc:1196] T 14896b5ab8194863ab12f17d9075944b P 5e9007ef0cd447bbae5844cef38ba52c: Deleting WAL directory at /tmp/dist-test-taskUci3DU/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781966829091300-388-0/minicluster-data/ts-0/wal/wals/14896b5ab8194863ab12f17d9075944b
I20260620 14:49:35.881965 12268 ts_tablet_manager.cc:1939] T 14896b5ab8194863ab12f17d9075944b P 5e9007ef0cd447bbae5844cef38ba52c: Deleting consensus metadata
I20260620 14:49:35.883917 11721 catalog_manager.cc:4841] TS 5e9007ef0cd447bbae5844cef38ba52c (127.0.97.1:43109): tablet 14896b5ab8194863ab12f17d9075944b (table test-table [id=1cb553e100104d97a319c83585c560ed]) successfully deleted
I20260620 14:49:35.888262 12240 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:49:35.853453 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44008 (local address 127.0.97.1:43109)
0620 14:49:35.853697 (+ 244us) server_negotiation.cc:206] Beginning negotiation
0620 14:49:35.853706 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0620 14:49:35.868193 (+ 14487us) server_negotiation.cc:407] Connection header received
0620 14:49:35.868397 (+ 204us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:49:35.868403 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:49:35.868738 (+ 335us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:49:35.868862 (+ 124us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:49:35.870514 (+ 1652us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:49:35.871774 (+ 1260us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:49:35.874229 (+ 2455us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:49:35.874859 (+ 630us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:49:35.875007 (+ 148us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:49:35.880478 (+ 5471us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 14:49:35.880495 (+ 17us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 14:49:35.880516 (+ 21us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 14:49:35.880569 (+ 53us) server_negotiation.cc:922] calling sasl_server_start()
0620 14:49:35.884823 (+ 4254us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:49:35.885604 (+ 781us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:49:35.885617 (+ 13us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:49:35.885621 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:49:35.885788 (+ 167us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:49:35.886156 (+ 368us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:49:35.886161 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:49:35.886165 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:49:35.886636 (+ 471us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 14:49:35.887379 (+ 743us) server_negotiation.cc:1035] Waiting for connection context
0620 14:49:35.887562 (+ 183us) server_negotiation.cc:299] Negotiation successful
0620 14:49:35.887846 (+ 284us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":71}
I20260620 14:49:35.888954 11943 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 14:49:35.981400 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskUci3DU/build/asan/bin/kudu with pid 11815
I20260620 14:49:35.999298 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskUci3DU/build/asan/bin/kudu pid 11954
Jun 20 14:49:36 dist-test-slave-q9j6 krb5kdc[11259](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781966971, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.2@KRBTEST.COM
I20260620 14:49:36.017027 12241 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:49:36.000170 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37786 (local address 127.0.97.2:32829)
0620 14:49:36.000365 (+ 195us) server_negotiation.cc:206] Beginning negotiation
0620 14:49:36.000372 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0620 14:49:36.000394 (+ 22us) server_negotiation.cc:407] Connection header received
0620 14:49:36.000522 (+ 128us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:49:36.000528 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:49:36.000626 (+ 98us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:49:36.000749 (+ 123us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:49:36.001972 (+ 1223us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:49:36.003194 (+ 1222us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:49:36.004783 (+ 1589us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:49:36.005531 (+ 748us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:49:36.005676 (+ 145us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:49:36.010352 (+ 4676us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 14:49:36.010369 (+ 17us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 14:49:36.010386 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 14:49:36.010423 (+ 37us) server_negotiation.cc:922] calling sasl_server_start()
0620 14:49:36.013684 (+ 3261us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:49:36.014673 (+ 989us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:49:36.014680 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:49:36.014682 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:49:36.014844 (+ 162us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:49:36.015305 (+ 461us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:49:36.015309 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:49:36.015311 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:49:36.015762 (+ 451us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 14:49:36.016016 (+ 254us) server_negotiation.cc:1035] Waiting for connection context
0620 14:49:36.016390 (+ 374us) server_negotiation.cc:299] Negotiation successful
0620 14:49:36.016606 (+ 216us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":76}
I20260620 14:49:36.017756 12086 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 14:49:36.133386 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskUci3DU/build/asan/bin/kudu with pid 11954
I20260620 14:49:36.151636 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskUci3DU/build/asan/bin/kudu pid 12097
Jun 20 14:49:36 dist-test-slave-q9j6 krb5kdc[11259](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781966971, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.3@KRBTEST.COM
I20260620 14:49:36.167110 12242 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:49:36.152461 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34514 (local address 127.0.97.3:36407)
0620 14:49:36.152637 (+ 176us) server_negotiation.cc:206] Beginning negotiation
0620 14:49:36.152642 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 14:49:36.152661 (+ 19us) server_negotiation.cc:407] Connection header received
0620 14:49:36.152822 (+ 161us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:49:36.152826 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:49:36.152899 (+ 73us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:49:36.152987 (+ 88us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:49:36.154257 (+ 1270us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:49:36.154970 (+ 713us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:49:36.156376 (+ 1406us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:49:36.156757 (+ 381us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:49:36.156855 (+ 98us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:49:36.161264 (+ 4409us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 14:49:36.161280 (+ 16us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 14:49:36.161296 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 14:49:36.161334 (+ 38us) server_negotiation.cc:922] calling sasl_server_start()
0620 14:49:36.164285 (+ 2951us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:49:36.165089 (+ 804us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:49:36.165097 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:49:36.165099 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:49:36.165228 (+ 129us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:49:36.165645 (+ 417us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:49:36.165651 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:49:36.165657 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:49:36.166032 (+ 375us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 14:49:36.166242 (+ 210us) server_negotiation.cc:1035] Waiting for connection context
0620 14:49:36.166547 (+ 305us) server_negotiation.cc:299] Negotiation successful
0620 14:49:36.166709 (+ 162us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":68}
I20260620 14:49:36.167961 12220 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 14:49:36.280136 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskUci3DU/build/asan/bin/kudu with pid 12097
I20260620 14:49:36.296624 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskUci3DU/build/asan/bin/kudu pid 11703
I20260620 14:49:36.297978 11765 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 14:49:36.410032 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskUci3DU/build/asan/bin/kudu with pid 11703
I20260620 14:49:36.432750 388 mini_ranger_kms.cc:60] Stopping Ranger KMS...
I20260620 14:49:36.778494 388 mini_ranger_kms.cc:62] Stopped Ranger KMS
I20260620 14:49:36.778682 388 mini_ranger.cc:66] Stopping Ranger...
I20260620 14:49:37.124815 388 mini_ranger.cc:68] Stopped Ranger
2026-06-20 14:49:37.125 UTC [11289] LOG: received smart shutdown request
2026-06-20 14:49:37.129 UTC [11289] LOG: background worker "logical replication launcher" (PID 11298) exited with exit code 1
2026-06-20 14:49:37.129 UTC [11293] LOG: shutting down
2026-06-20 14:49:37.297 UTC [11289] LOG: database system is shut down
2026-06-20T14:49:37Z chronyd exiting
[ OK ] SecurityITest.TestEncryptionWithKMSIntegration (48535 ms)
[ RUN ] SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers
Loading random data
Initializing database '/tmp/dist-test-taskUci3DU/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781966829091300-388-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 20 14:49:37 dist-test-slave-q9j6 krb5kdc[12284](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 20 14:49:37 dist-test-slave-q9j6 krb5kdc[12284](info): set up 2 sockets
Jun 20 14:49:37 dist-test-slave-q9j6 krb5kdc[12284](info): commencing operation
krb5kdc: starting...
W20260620 14:49:38.536011 388 mini_kdc.cc:121] Time spent starting KDC: real 1.071s user 0.003s sys 0.020s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskUci3DU/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781966829091300-388-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskUci3DU/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781966829091300-388-0/krb5kdc/test-user.keytab.
Jun 20 14:49:38 dist-test-slave-q9j6 krb5kdc[12284](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781966978, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-20T14:49:38Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T14:49:38Z Disabled control of system clock
WARNING: no policy specified for rangeradmin/127.0.97.20@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangeradmin/127.0.97.20@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangeradmin/127.0.97.20@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskUci3DU/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781966829091300-388-0/krb5kdc/rangeradmin_127.0.97.20@KRBTEST.COM.keytab.
Entry for principal rangeradmin/127.0.97.20@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskUci3DU/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781966829091300-388-0/krb5kdc/rangeradmin_127.0.97.20@KRBTEST.COM.keytab.
WARNING: no policy specified for rangerlookup/127.0.97.20@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangerlookup/127.0.97.20@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangerlookup/127.0.97.20@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskUci3DU/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781966829091300-388-0/krb5kdc/rangerlookup_127.0.97.20@KRBTEST.COM.keytab.
Entry for principal rangerlookup/127.0.97.20@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskUci3DU/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781966829091300-388-0/krb5kdc/rangerlookup_127.0.97.20@KRBTEST.COM.keytab.
WARNING: no policy specified for HTTP/127.0.97.20@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.0.97.20@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.0.97.20@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskUci3DU/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781966829091300-388-0/krb5kdc/HTTP_127.0.97.20@KRBTEST.COM.keytab.
Entry for principal HTTP/127.0.97.20@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskUci3DU/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781966829091300-388-0/krb5kdc/HTTP_127.0.97.20@KRBTEST.COM.keytab.
I20260620 14:49:38.864689 388 mini_postgres.cc:62] Running initdb...
The files belonging to this database system will be owned by user "slave".
This user must also own the server process.
The database cluster will be initialized with locale "C".
The default database encoding has accordingly been set to "SQL_ASCII".
The default text search configuration will be set to "english".
Data page checksums are disabled.
creating directory /tmp/dist-test-taskUci3DU/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781966829091300-388-0/minicluster-data/postgres ... ok
creating subdirectories ... ok
selecting dynamic shared memory implementation ... posix
selecting default max_connections ... 100
selecting default shared_buffers ... 128MB
selecting default time zone ... Etc/UTC
creating configuration files ... ok
running bootstrap script ... ok
performing post-bootstrap initialization ... ok
syncing data to disk ... ok
initdb: warning: enabling "trust" authentication for local connections
You can change this by editing pg_hba.conf or using the option -A, or
--auth-local and --auth-host, the next time you run initdb.
Success. You can now start the database server using:
/tmp/dist-test-taskUci3DU/build/asan/bin/postgres/pg_ctl -D /tmp/dist-test-taskUci3DU/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781966829091300-388-0/minicluster-data/postgres -l logfile start
2026-06-20 14:49:41.246 UTC [12314] LOG: starting PostgreSQL 12.2 on x86_64-pc-linux-gnu, compiled by cc (GCC) 8.3.1 20190311 (Red Hat 8.3.1-3), 64-bit
2026-06-20 14:49:41.246 UTC [12314] LOG: listening on IPv4 address "127.0.97.20", port 56421
2026-06-20 14:49:41.250 UTC [12314] LOG: listening on Unix socket "/tmp/.s.PGSQL.56421"
2026-06-20 14:49:41.270 UTC [12317] LOG: database system was shut down at 2026-06-20 14:49:39 UTC
2026-06-20 14:49:41.274 UTC [12314] LOG: database system is ready to accept connections
I20260620 14:49:42.251672 388 mini_postgres.cc:96] Postgres bound to 56421
2026-06-20 14:49:42.268 UTC [12325] FATAL: database "slave" does not exist
127.0.97.20:56421 - accepting connections
I20260620 14:49:42.269173 388 mini_ranger.cc:161] Starting Ranger...
I20260620 14:49:42.295456 388 mini_ranger.cc:84] Created miniranger Postgres user
I20260620 14:49:42.737270 388 mini_ranger.cc:87] Created ranger Postgres database
I20260620 14:49:42.737391 388 mini_ranger.cc:178] Starting Ranger out of /tmp/dist-test-taskUci3DU/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781966829091300-388-0/minicluster-data/ranger-admin
2026-06-20 14:49:43,026 [I] DB FLAVOR :POSTGRES
2026-06-20 14:49:43,027 [I] --------- Verifying Ranger DB connection ---------
2026-06-20 14:49:43,027 [I] Checking connection..
2026-06-20 14:49:43,027 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskUci3DU/build/asan/bin/postgresql.jar:/tmp/dist-test-taskUci3DU/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.97.20:56421/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select 1;"
2026-06-20 14:49:43,299 [I] Checking connection passed.
2026-06-20 14:49:43,299 [I] --------- Verifying version history table ---------
2026-06-20 14:49:43,299 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskUci3DU/build/asan/bin/postgresql.jar:/tmp/dist-test-taskUci3DU/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.97.20:56421/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-06-20 14:49:43,550 [I] Table x_db_version_h does not exist in database ranger
2026-06-20 14:49:43,551 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskUci3DU/build/asan/bin/postgresql.jar:/tmp/dist-test-taskUci3DU/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.97.20:56421/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-06-20 14:49:43,794 [I] Table x_db_version_h does not exist in database ranger
2026-06-20 14:49:43,794 [I] Importing x_db_version_h table schema to database ranger from file: create_dbversion_catalog.sql
2026-06-20 14:49:43,794 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskUci3DU/build/asan/bin/postgresql.jar:/tmp/dist-test-taskUci3DU/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.97.20:56421/ranger -u miniranger -p '********' -noheader -trim -c \; -input /tmp/dist-test-taskUci3DU/thirdparty/src/ranger-2.1.0-admin/db/postgres/create_dbversion_catalog.sql
2026-06-20 14:49:44.057 UTC [12389] WARNING: there is no transaction in progress
2026-06-20 14:49:44,063 [I] create_dbversion_catalog.sql file imported successfully
2026-06-20 14:49:44,064 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskUci3DU/build/asan/bin/postgresql.jar:/tmp/dist-test-taskUci3DU/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.97.20:56421/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-06-20 14:49:44,299 [I] Table x_db_version_h already exists in database 'ranger'
2026-06-20 14:49:44,300 [I] --------- Importing Ranger Core DB Schema ---------
2026-06-20 14:49:44,300 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskUci3DU/build/asan/bin/postgresql.jar:/tmp/dist-test-taskUci3DU/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.97.20:56421/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'CORE_DB_SCHEMA' and active = 'Y';"
2026-06-20 14:49:44,562 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskUci3DU/build/asan/bin/postgresql.jar:/tmp/dist-test-taskUci3DU/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.97.20:56421/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'CORE_DB_SCHEMA' and active = 'N';"
2026-06-20 14:49:44,807 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskUci3DU/build/asan/bin/postgresql.jar:/tmp/dist-test-taskUci3DU/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.97.20:56421/ranger -u miniranger -p '********' -noheader -trim -c \; -query "insert into x_db_version_h (version, inst_at, inst_by, updated_at, updated_by,active) values ('CORE_DB_SCHEMA', current_timestamp, 'Ranger 2.1.0', current_timestamp, 'dist-test-slave-q9j6.c.gcp-upstream.internal','N') ;"
2026-06-20 14:49:45,043 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskUci3DU/build/asan/bin/postgresql.jar:/tmp/dist-test-taskUci3DU/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.97.20:56421/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_portal_user') as temp;"
2026-06-20 14:49:45,296 [I] Table x_portal_user does not exist in database ranger
2026-06-20 14:49:45,296 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskUci3DU/build/asan/bin/postgresql.jar:/tmp/dist-test-taskUci3DU/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.97.20:56421/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_policy_ref_group') as temp;"
2026-06-20 14:49:45,550 [I] Table x_policy_ref_group does not exist in database ranger
2026-06-20 14:49:45,550 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskUci3DU/build/asan/bin/postgresql.jar:/tmp/dist-test-taskUci3DU/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.97.20:56421/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and active = 'Y';"
2026-06-20 14:49:45,817 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskUci3DU/build/asan/bin/postgresql.jar:/tmp/dist-test-taskUci3DU/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.97.20:56421/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'JAVA_PATCHES' and active = 'Y';"
2026-06-20 14:49:46,058 [I] Importing DB schema to database ranger from file: ranger_core_db_postgres.sql
2026-06-20 14:49:46,059 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskUci3DU/build/asan/bin/postgresql.jar:/tmp/dist-test-taskUci3DU/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.97.20:56421/ranger -u miniranger -p '********' -noheader -trim -c \; -input /tmp/dist-test-taskUci3DU/thirdparty/src/ranger-2.1.0-admin/db/postgres/optimized/current/ranger_core_db_postgres.sql
2026-06-20 14:49:47.040 UTC [12497] WARNING: there is no transaction in progress
2026-06-20 14:49:47.057 UTC [12497] WARNING: there is no transaction in progress
2026-06-20 14:49:47.073 UTC [12497] WARNING: there is no transaction in progress
2026-06-20 14:49:47.088 UTC [12497] WARNING: there is no transaction in progress
2026-06-20 14:49:47.103 UTC [12497] WARNING: there is no transaction in progress
2026-06-20 14:49:47.121 UTC [12497] WARNING: there is no transaction in progress
2026-06-20 14:49:47.176 UTC [12497] WARNING: there is no transaction in progress
2026-06-20 14:49:47.183 UTC [12497] WARNING: there is no transaction in progress
2026-06-20 14:49:47.197 UTC [12497] WARNING: there is no transaction in progress
2026-06-20 14:49:47.208 UTC [12497] WARNING: there is no transaction in progress
2026-06-20 14:49:47.218 UTC [12497] WARNING: there is no transaction in progress
2026-06-20 14:49:47.233 UTC [12497] WARNING: there is no transaction in progress
2026-06-20 14:49:47.243 UTC [12497] WARNING: there is no transaction in progress
2026-06-20 14:49:47.249 UTC [12497] WARNING: there is no transaction in progress
2026-06-20 14:49:47.256 UTC [12497] WARNING: there is no transaction in progress
2026-06-20 14:49:47.262 UTC [12497] WARNING: there is no transaction in progress
2026-06-20 14:49:47,852 [I] ranger_core_db_postgres.sql file imported successfully
2026-06-20 14:49:47,852 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskUci3DU/build/asan/bin/postgresql.jar:/tmp/dist-test-taskUci3DU/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.97.20:56421/ranger -u miniranger -p '********' -noheader -trim -c \; -query "update x_db_version_h set inst_by='Ranger 2.1.0' where active='Y' and updated_by='localhost';"
2026-06-20 14:49:48,078 [I] Patches status entries updated from base ranger version to current installed ranger version:Ranger 2.1.0
2026-06-20 14:49:48,078 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskUci3DU/build/asan/bin/postgresql.jar:/tmp/dist-test-taskUci3DU/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.97.20:56421/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_portal_user') as temp;"
2026-06-20 14:49:48,325 [I] Table x_portal_user already exists in database 'ranger'
2026-06-20 14:49:48,325 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskUci3DU/build/asan/bin/postgresql.jar:/tmp/dist-test-taskUci3DU/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.97.20:56421/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_policy_ref_group') as temp;"
2026-06-20 14:49:48,570 [I] Table x_policy_ref_group already exists in database 'ranger'
2026-06-20 14:49:48,570 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskUci3DU/build/asan/bin/postgresql.jar:/tmp/dist-test-taskUci3DU/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.97.20:56421/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and active = 'Y';"
2026-06-20 14:49:48,826 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskUci3DU/build/asan/bin/postgresql.jar:/tmp/dist-test-taskUci3DU/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.97.20:56421/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'JAVA_PATCHES' and active = 'Y';"
2026-06-20 14:49:49,068 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskUci3DU/build/asan/bin/postgresql.jar:/tmp/dist-test-taskUci3DU/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.97.20:56421/ranger -u miniranger -p '********' -noheader -trim -c \; -query "update x_db_version_h set active='Y' where version='CORE_DB_SCHEMA' and active='N' and updated_by='dist-test-slave-q9j6.c.gcp-upstream.internal';"
2026-06-20 14:49:49,320 [I] CORE_DB_SCHEMA import status has been updated
2026-06-20 14:49:49,321 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskUci3DU/build/asan/bin/postgresql.jar:/tmp/dist-test-taskUci3DU/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.97.20:56421/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and inst_by = 'Ranger 2.1.0' and active = 'Y';"
2026-06-20 14:49:49,564 [I] DB_PATCHES have already been applied
I20260620 14:49:49.569977 388 mini_ranger.cc:191] Using Ranger class path: /tmp/dist-test-taskUci3DU/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781966829091300-388-0/minicluster-data/ranger-admin:/tmp/dist-test-taskUci3DU/thirdparty/src/ranger-2.1.0-admin/ews/lib/*:/usr/lib/jvm/java-1.8.0-openjdk-amd64/lib/*:/tmp/dist-test-taskUci3DU/thirdparty/src/hadoop-3.2.0/*:/tmp/dist-test-taskUci3DU/build/asan/bin/postgresql.jar:/tmp/dist-test-taskUci3DU/thirdparty/src/ranger-2.1.0-admin/ews/webapp
I20260620 14:49:49.570063 388 mini_ranger.cc:193] Using host: 127.0.97.20
I20260620 14:49:49.584894 388 mini_ranger.cc:227] Ranger admin URL: http://127.0.97.20:34203
Jun 20, 2026 2:49:49 PM org.apache.ranger.server.tomcat.EmbeddedServer getKeyManagers
WARNING: Config 'ranger.keystore.file' or 'ranger.service.https.attrib.keystore.file' is not found or contains blank value
Jun 20, 2026 2:49:49 PM org.apache.ranger.server.tomcat.EmbeddedServer getTrustManagers
WARNING: Config 'ranger.truststore.file' is not found or contains blank value!
Jun 20, 2026 2:49:50 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Deriving webapp folder from catalina.base property. folder=/tmp/dist-test-taskUci3DU/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781966829091300-388-0/minicluster-data/ranger-admin/ews/webapp
Jun 20, 2026 2:49:50 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Webapp file =/tmp/dist-test-taskUci3DU/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781966829091300-388-0/minicluster-data/ranger-admin/ews/webapp, webAppName = /
Jun 20, 2026 2:49:50 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Adding webapp [/] = path [/tmp/dist-test-taskUci3DU/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781966829091300-388-0/minicluster-data/ranger-admin/ews/webapp] .....
Jun 20, 2026 2:49:50 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Finished init of webapp [/] = path [/tmp/dist-test-taskUci3DU/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781966829091300-388-0/minicluster-data/ranger-admin/ews/webapp].
Jun 20, 2026 2:49:50 PM org.apache.ranger.server.tomcat.EmbeddedServer startServer
INFO: Server Name : miniranger
I20260620 14:49:50.712837 388 mini_ranger.cc:160] Time spent starting Ranger: real 8.444s user 0.006s sys 0.034s
Jun 20, 2026 2:50:10 PM com.sun.jersey.api.core.PackagesResourceConfig init
INFO: Scanning for root resource and provider classes in the packages:
org.apache.ranger.rest
org.apache.ranger.common
xa.rest
Jun 20, 2026 2:50:11 PM com.sun.jersey.api.core.ScanningResourceConfig logClasses
INFO: Root resource classes found:
class org.apache.ranger.rest.XAuditREST
class org.apache.ranger.rest.TagREST
class org.apache.ranger.rest.UserREST
class org.apache.ranger.rest.RoleREST
class org.apache.ranger.rest.MetricsREST
class org.apache.ranger.rest.SecurityZoneREST
class org.apache.ranger.rest.ServiceREST
class org.apache.ranger.rest.AssetREST
class org.apache.ranger.rest.PublicAPIsv2
class org.apache.ranger.rest.PublicAPIs
class org.apache.ranger.rest.XUserREST
class org.apache.ranger.rest.XKeyREST
Jun 20, 2026 2:50:11 PM com.sun.jersey.api.core.ScanningResourceConfig logClasses
INFO: Provider classes found:
class org.apache.ranger.common.RangerJAXBContextResolver
Jun 20, 2026 2:50:11 PM com.sun.jersey.spi.spring.container.servlet.SpringServlet getContext
INFO: Using default applicationContext
Jun 20, 2026 2:50:11 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, assetREST, of type org.apache.ranger.rest.AssetREST as a root resource class
Jun 20, 2026 2:50:11 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, metricsREST, of type org.apache.ranger.rest.MetricsREST as a root resource class
Jun 20, 2026 2:50:11 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, publicAPIs, of type org.apache.ranger.rest.PublicAPIs as a root resource class
Jun 20, 2026 2:50:11 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, publicAPIsv2, of type org.apache.ranger.rest.PublicAPIsv2 as a root resource class
Jun 20, 2026 2:50:11 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, roleREST, of type org.apache.ranger.rest.RoleREST as a root resource class
Jun 20, 2026 2:50:11 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, securityZoneREST, of type org.apache.ranger.rest.SecurityZoneREST as a root resource class
Jun 20, 2026 2:50:11 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, serviceREST, of type org.apache.ranger.rest.ServiceREST as a root resource class
Jun 20, 2026 2:50:11 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, tagREST, of type org.apache.ranger.rest.TagREST as a root resource class
Jun 20, 2026 2:50:11 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, userREST, of type org.apache.ranger.rest.UserREST as a root resource class
Jun 20, 2026 2:50:11 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XAuditREST, of type org.apache.ranger.rest.XAuditREST as a root resource class
Jun 20, 2026 2:50:11 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XKeyREST, of type org.apache.ranger.rest.XKeyREST as a root resource class
Jun 20, 2026 2:50:11 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XUserREST, of type org.apache.ranger.rest.XUserREST as a root resource class
Jun 20, 2026 2:50:11 PM com.sun.jersey.server.impl.application.WebApplicationImpl _initiate
INFO: Initiating Jersey application, version 'Jersey: 1.19.3 10/24/2016 03:58 PM'
Jun 20, 2026 2:50:12 PM com.sun.jersey.spi.inject.Errors processErrorMessages
WARNING: The following warnings have been detected with resource and/or provider classes:
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInExcel(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse), MUST return a non-void type.
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInJson(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse,java.lang.Boolean), MUST return a non-void type.
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInCsv(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse) throws java.io.IOException, MUST return a non-void type.
I20260620 14:50:12.687376 388 mini_ranger.cc:261] Created Kudu service
WARNING: no policy specified for rangerkms/127.0.97.20@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangerkms/127.0.97.20@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangerkms/127.0.97.20@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskUci3DU/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781966829091300-388-0/krb5kdc/rangerkms_127.0.97.20@KRBTEST.COM.keytab.
Entry for principal rangerkms/127.0.97.20@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskUci3DU/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781966829091300-388-0/krb5kdc/rangerkms_127.0.97.20@KRBTEST.COM.keytab.
WARNING: no policy specified for HTTP/127.0.97.20@KRBTEST.COM; defaulting to no policy
add_principal: Principal or policy already exists while creating "HTTP/127.0.97.20@KRBTEST.COM".
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.0.97.20@KRBTEST.COM with kvno 3, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskUci3DU/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781966829091300-388-0/krb5kdc/HTTP_127.0.97.20@KRBTEST.COM.keytab.
Entry for principal HTTP/127.0.97.20@KRBTEST.COM with kvno 3, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskUci3DU/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781966829091300-388-0/krb5kdc/HTTP_127.0.97.20@KRBTEST.COM.keytab.
WARNING: no policy specified for keyadmin@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "keyadmin@KRBTEST.COM" created.
Jun 20 14:50:12 dist-test-slave-q9j6 krb5kdc[12284](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781967012, etypes {rep=17 tkt=17 ses=17}, keyadmin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for keyadmin@KRBTEST.COM:
I20260620 14:50:12.860205 388 mini_ranger_kms.cc:208] Starting Ranger KMS...
I20260620 14:50:12.881660 388 mini_ranger_kms.cc:78] Created minirangerkms Postgres user
I20260620 14:50:13.680548 388 mini_ranger_kms.cc:81] Created rangerkms Postgres database
I20260620 14:50:13.680673 388 mini_ranger_kms.cc:226] Starting Ranger KMS out of /tmp/dist-test-taskUci3DU/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781966829091300-388-0/minicluster-data/ranger-kms
I20260620 14:50:13.680719 388 mini_ranger_kms.cc:227] Using postgres at 127.0.97.20:56421
2026-06-20 14:50:14,038 [I] DB FLAVOR :POSTGRES
2026-06-20 14:50:14,038 [I] --------- Verifying Ranger DB connection ---------
2026-06-20 14:50:14,038 [I] Checking connection
2026-06-20 14:50:14,039 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskUci3DU/build/asan/bin/postgresql.jar:/tmp/dist-test-taskUci3DU/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781966829091300-388-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.97.20:56421/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -query "SELECT 1;"
2026-06-20 14:50:14,291 [I] connection success
2026-06-20 14:50:14,291 [I] --------- Verifying Ranger DB tables ---------
2026-06-20 14:50:14,291 [I] Verifying table ranger_masterkey in database rangerkms
2026-06-20 14:50:14,291 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskUci3DU/build/asan/bin/postgresql.jar:/tmp/dist-test-taskUci3DU/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781966829091300-388-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.97.20:56421/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='rangerkms' and table_name = 'ranger_masterkey') as temp;"
2026-06-20 14:50:14,542 [I] Table ranger_masterkey does not exist in database rangerkms
2026-06-20 14:50:14,542 [I] --------- Importing Ranger Core DB Schema ---------
2026-06-20 14:50:14,542 [I] Importing db schema to database rangerkms from file: kms_core_db_postgres.sql
2026-06-20 14:50:14,542 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskUci3DU/build/asan/bin/postgresql.jar:/tmp/dist-test-taskUci3DU/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781966829091300-388-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.97.20:56421/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -input /tmp/dist-test-taskUci3DU/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781966829091300-388-0/minicluster-data/ranger-kms/db/postgres/kms_core_db_postgres.sql
2026-06-20 14:50:14,818 [I] kms_core_db_postgres.sql DB schema imported successfully
I20260620 14:50:15.280612 388 mini_ranger_kms.cc:310] Created kms service
I20260620 14:50:15.532624 388 mini_ranger_kms.cc:326] Created kudu user
I20260620 14:50:15.582484 388 mini_ranger_kms.cc:343] Created rangerkms user
Jun 20 14:50:15 dist-test-slave-q9j6 krb5kdc[12284](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781967012, etypes {rep=17 tkt=17 ses=17}, keyadmin@KRBTEST.COM for HTTP/127.0.97.20@KRBTEST.COM
I20260620 14:50:16.008790 388 mini_ranger_kms.cc:384] Added ranger policy
I20260620 14:50:16.009132 388 mini_ranger_kms.cc:238] Using RangerKMS classpath: /tmp/dist-test-taskUci3DU/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781966829091300-388-0/minicluster-data/ranger-kms:/tmp/dist-test-taskUci3DU/build/asan/bin/postgresql.jar:/tmp/dist-test-taskUci3DU/thirdparty/src/ranger-2.1.0-kms/ews/webapp/WEB-INF/classes/lib/*:/tmp/dist-test-taskUci3DU/thirdparty/src/ranger-2.1.0-kms/ews/webapp/lib/*:/usr/lib/jvm/java-1.8.0-openjdk-amd64/lib/*
I20260620 14:50:16.009200 388 mini_ranger_kms.cc:240] Using host: 127.0.97.20
I20260620 14:50:16.026556 388 mini_ranger_kms.cc:276] Ranger KMS PID: 12677
I20260620 14:50:16.026701 388 mini_ranger_kms.cc:277] Ranger KMS URL: http://127.0.97.20:51495
Jun 20, 2026 2:50:17 PM org.apache.ranger.server.tomcat.EmbeddedServer getKeyManagers
WARNING: Config 'ranger.keystore.file' or 'ranger.service.https.attrib.keystore.file' is not found or contains blank value
Jun 20, 2026 2:50:17 PM org.apache.ranger.server.tomcat.EmbeddedServer getTrustManagers
WARNING: Config 'ranger.truststore.file' is not found or contains blank value!
Jun 20, 2026 2:50:17 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Webapp file =/tmp/dist-test-taskUci3DU/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781966829091300-388-0/minicluster-data/ranger-kms/ews/webapp, webAppName = /kms
Jun 20, 2026 2:50:17 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Adding webapp [/kms] = path [/tmp/dist-test-taskUci3DU/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781966829091300-388-0/minicluster-data/ranger-kms/ews/webapp] .....
Jun 20, 2026 2:50:17 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Finished init of webapp [/kms] = path [/tmp/dist-test-taskUci3DU/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781966829091300-388-0/minicluster-data/ranger-kms/ews/webapp].
Jun 20, 2026 2:50:17 PM org.apache.ranger.server.tomcat.EmbeddedServer startServer
INFO: Server Name : minirangerkms
Jun 20 14:50:18 dist-test-slave-q9j6 krb5kdc[12284](info): AS_REQ (1 etypes {17}) 127.0.0.1: ISSUE: authtime 1781967018, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.0.97.20@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 14:50:18.348150 388 mini_ranger_kms.cc:207] Time spent starting Ranger KMS: real 5.488s user 0.068s sys 0.305s
I20260620 14:50:18.348403 388 mini_ranger_kms.cc:397] {"name":"kuduclusterkey","cipher":"AES/CTR/NoPadding","length":128,"description":"kuduclusterkey"}
I20260620 14:50:18.348582 388 mini_ranger_kms.cc:401] 127.0.97.20:51495/kms/v1/keys
Jun 20 14:50:18 dist-test-slave-q9j6 krb5kdc[12284](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781967018, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.0.97.20@KRBTEST.COM for HTTP/127.0.97.20@KRBTEST.COM
Jun 20 14:50:19 dist-test-slave-q9j6 krb5kdc[12284](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781967018, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.0.97.20@KRBTEST.COM for HTTP/127.0.97.20@KRBTEST.COM
Jun 20 14:50:21 dist-test-slave-q9j6 krb5kdc[12284](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781967021, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
WARNING: no policy specified for kudu/127.0.97.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskUci3DU/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781966829091300-388-0/krb5kdc/kudu_127.0.97.62.keytab.
Entry for principal kudu/127.0.97.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskUci3DU/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781966829091300-388-0/krb5kdc/kudu_127.0.97.62.keytab.
I20260620 14:50:21.369371 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskUci3DU/build/asan/bin/kudu
/tmp/dist-test-taskUci3DU/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskUci3DU/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781966829091300-388-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskUci3DU/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781966829091300-388-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskUci3DU/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781966829091300-388-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskUci3DU/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781966829091300-388-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:38829
--webserver_interface=127.0.97.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskUci3DU/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781966829091300-388-0/krb5kdc/kudu_127.0.97.62.keytab
--principal=kudu/127.0.97.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:34389
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.97.62:38829
--ranger_config_path=/tmp/dist-test-taskUci3DU/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781966829091300-388-0/minicluster-data/ranger-client
--trusted_user_acl=test-admin
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.0.97.20:51495/kms
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskUci3DU/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781966829091300-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskUci3DU/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781966829091300-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskUci3DU/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781966829091300-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 14:50:21.713707 12728 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 14:50:21.714072 12728 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 14:50:21.714175 12728 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 14:50:21.721583 12728 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 14:50:21.721704 12728 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 14:50:21.721748 12728 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 14:50:21.721797 12728 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 14:50:21.721845 12728 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 14:50:21.732264 12728 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:34389
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskUci3DU/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781966829091300-388-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskUci3DU/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781966829091300-388-0/minicluster-data/master-0/wal
--ranger_kms_url=127.0.97.20:51495/kms
--trusted_user_acl=<redacted>
--ipki_ca_key_size=768
--master_addresses=127.0.97.62:38829
--ranger_config_path=/tmp/dist-test-taskUci3DU/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781966829091300-388-0/minicluster-data/ranger-client
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskUci3DU/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781966829091300-388-0/krb5kdc/kudu_127.0.97.62.keytab
--principal=kudu/127.0.97.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:38829
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskUci3DU/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781966829091300-388-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.97.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskUci3DU/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781966829091300-388-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 13:41:20 UTC on ccycloud.quasar-vdagoi.root.comops.site
build id 4457462
ASAN enabled
I20260620 14:50:21.733567 12728 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 14:50:21.734859 12728 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 14:50:21.747337 12733 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 14:50:21.747409 12734 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 14:50:21.747378 12736 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 14:50:21.747669 12728 server_base.cc:1034] running on GCE node
I20260620 14:50:21.748706 12728 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 14:50:21.750238 12728 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 14:50:21.751495 12728 hybrid_clock.cc:648] HybridClock initialized: now 1781967021751447 us; error 40 us; skew 500 ppm
I20260620 14:50:21.751896 12728 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 14:50:21 dist-test-slave-q9j6 krb5kdc[12284](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781967021, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 14:50:21.759270 12728 init.cc:377] Logged in from keytab as kudu/127.0.97.62@KRBTEST.COM (short username kudu)
I20260620 14:50:21.760974 12728 webserver.cc:460] Webserver started at http://127.0.97.62:33863/ using document root <none> and password file <none>
I20260620 14:50:21.761715 12728 fs_manager.cc:359] Metadata directory not provided
I20260620 14:50:21.761818 12728 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 14:50:21.762090 12728 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Jun 20 14:50:21 dist-test-slave-q9j6 krb5kdc[12284](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781967021, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.62@KRBTEST.COM for HTTP/127.0.97.20@KRBTEST.COM
I20260620 14:50:21.824388 12728 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskUci3DU/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781966829091300-388-0/minicluster-data/master-0/data/instance:
uuid: "384f5045b6a1446b8de95d923f9e10c2"
format_stamp: "Formatted at 2026-06-20 14:50:21 on dist-test-slave-q9j6"
server_key: "d06409155333808df639af13e3ed2e42"
server_key_iv: "621d8741b12cd60fe34d912ab3cd141f"
server_key_version: "kuduclusterkey@0"
I20260620 14:50:21.825485 12728 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskUci3DU/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781966829091300-388-0/minicluster-data/master-0/wal/instance:
uuid: "384f5045b6a1446b8de95d923f9e10c2"
format_stamp: "Formatted at 2026-06-20 14:50:21 on dist-test-slave-q9j6"
server_key: "d06409155333808df639af13e3ed2e42"
server_key_iv: "621d8741b12cd60fe34d912ab3cd141f"
server_key_version: "kuduclusterkey@0"
I20260620 14:50:21.832870 12728 fs_manager.cc:691] Time spent creating directory manager: real 0.007s user 0.008s sys 0.000s
I20260620 14:50:21.854647 12743 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 14:50:21.856551 12728 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.003s sys 0.000s
I20260620 14:50:21.856781 12728 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskUci3DU/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781966829091300-388-0/minicluster-data/master-0/data,/tmp/dist-test-taskUci3DU/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781966829091300-388-0/minicluster-data/master-0/wal
uuid: "384f5045b6a1446b8de95d923f9e10c2"
format_stamp: "Formatted at 2026-06-20 14:50:21 on dist-test-slave-q9j6"
server_key: "d06409155333808df639af13e3ed2e42"
server_key_iv: "621d8741b12cd60fe34d912ab3cd141f"
server_key_version: "kuduclusterkey@0"
I20260620 14:50:21.857057 12728 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskUci3DU/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781966829091300-388-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskUci3DU/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781966829091300-388-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskUci3DU/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781966829091300-388-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 14:50:21.896747 12728 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 14:50:21.901374 12728 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 14:50:21.901700 12728 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 14:50:21.921423 12728 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.62:38829
I20260620 14:50:21.921484 12794 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.62:38829 every 8 connection(s)
I20260620 14:50:21.922904 12728 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskUci3DU/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781966829091300-388-0/minicluster-data/master-0/data/info.pb
I20260620 14:50:21.924904 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskUci3DU/build/asan/bin/kudu as pid 12728
I20260620 14:50:21.925161 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskUci3DU/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781966829091300-388-0/minicluster-data/master-0/wal/instance
I20260620 14:50:21.925832 388 external_mini_cluster.cc:1428] Setting key fa4e233f7919aaa7dc138539c9c70468
I20260620 14:50:21.928058 12795 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 14:50:21.943877 12795 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 384f5045b6a1446b8de95d923f9e10c2: Bootstrap starting.
Jun 20 14:50:21 dist-test-slave-q9j6 krb5kdc[12284](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781967021, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260620 14:50:21.948170 12795 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 384f5045b6a1446b8de95d923f9e10c2: Neither blocks nor log segments found. Creating new log.
I20260620 14:50:21.949322 12795 log.cc:824] T 00000000000000000000000000000000 P 384f5045b6a1446b8de95d923f9e10c2: Log is configured to *not* fsync() on all Append() calls
I20260620 14:50:21.952575 12795 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 384f5045b6a1446b8de95d923f9e10c2: No bootstrap required, opened a new log
I20260620 14:50:21.956049 12798 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:50:21.928214 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40680 (local address 127.0.97.62:38829)
0620 14:50:21.934454 (+ 6240us) server_negotiation.cc:206] Beginning negotiation
0620 14:50:21.934468 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0620 14:50:21.934517 (+ 49us) server_negotiation.cc:407] Connection header received
0620 14:50:21.936306 (+ 1789us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:50:21.936337 (+ 31us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:50:21.936802 (+ 465us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:50:21.937176 (+ 374us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:50:21.938965 (+ 1789us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:50:21.940671 (+ 1706us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:50:21.942607 (+ 1936us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:50:21.943316 (+ 709us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:50:21.943466 (+ 150us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:50:21.948218 (+ 4752us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 14:50:21.948227 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 14:50:21.948243 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 14:50:21.948289 (+ 46us) server_negotiation.cc:922] calling sasl_server_start()
0620 14:50:21.951514 (+ 3225us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:50:21.952352 (+ 838us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:50:21.952359 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:50:21.952366 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:50:21.952529 (+ 163us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:50:21.952966 (+ 437us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:50:21.952977 (+ 11us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:50:21.952980 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:50:21.953552 (+ 572us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 14:50:21.953865 (+ 313us) server_negotiation.cc:1035] Waiting for connection context
0620 14:50:21.954384 (+ 519us) server_negotiation.cc:299] Negotiation successful
0620 14:50:21.954882 (+ 498us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":5958,"thread_start_us":5654,"threads_started":1}
I20260620 14:50:21.960523 12795 raft_consensus.cc:348] T 00000000000000000000000000000000 P 384f5045b6a1446b8de95d923f9e10c2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "384f5045b6a1446b8de95d923f9e10c2" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 38829 } }
I20260620 14:50:21.960986 12795 raft_consensus.cc:374] T 00000000000000000000000000000000 P 384f5045b6a1446b8de95d923f9e10c2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 14:50:21.961107 12795 raft_consensus.cc:729] T 00000000000000000000000000000000 P 384f5045b6a1446b8de95d923f9e10c2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 384f5045b6a1446b8de95d923f9e10c2, State: Initialized, Role: FOLLOWER
I20260620 14:50:21.961692 12795 consensus_queue.cc:260] T 00000000000000000000000000000000 P 384f5045b6a1446b8de95d923f9e10c2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "384f5045b6a1446b8de95d923f9e10c2" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 38829 } }
I20260620 14:50:21.961947 12795 raft_consensus.cc:388] T 00000000000000000000000000000000 P 384f5045b6a1446b8de95d923f9e10c2 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 14:50:21.962096 12795 raft_consensus.cc:482] T 00000000000000000000000000000000 P 384f5045b6a1446b8de95d923f9e10c2 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 14:50:21.962276 12795 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 384f5045b6a1446b8de95d923f9e10c2 [term 0 FOLLOWER]: Advancing to term 1
I20260620 14:50:21.964694 12795 raft_consensus.cc:504] T 00000000000000000000000000000000 P 384f5045b6a1446b8de95d923f9e10c2 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "384f5045b6a1446b8de95d923f9e10c2" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 38829 } }
I20260620 14:50:21.965353 12795 leader_election.cc:302] T 00000000000000000000000000000000 P 384f5045b6a1446b8de95d923f9e10c2 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 384f5045b6a1446b8de95d923f9e10c2; no voters:
I20260620 14:50:21.965965 12795 leader_election.cc:288] T 00000000000000000000000000000000 P 384f5045b6a1446b8de95d923f9e10c2 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260620 14:50:21.966130 12800 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 384f5045b6a1446b8de95d923f9e10c2 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 14:50:21.966909 12800 raft_consensus.cc:686] T 00000000000000000000000000000000 P 384f5045b6a1446b8de95d923f9e10c2 [term 1 LEADER]: Becoming Leader. State: Replica: 384f5045b6a1446b8de95d923f9e10c2, State: Running, Role: LEADER
I20260620 14:50:21.967439 12800 consensus_queue.cc:237] T 00000000000000000000000000000000 P 384f5045b6a1446b8de95d923f9e10c2 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "384f5045b6a1446b8de95d923f9e10c2" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 38829 } }
I20260620 14:50:21.968304 12795 sys_catalog.cc:564] T 00000000000000000000000000000000 P 384f5045b6a1446b8de95d923f9e10c2 [sys.catalog]: configured and running, proceeding with master startup.
I20260620 14:50:21.970434 12802 sys_catalog.cc:455] T 00000000000000000000000000000000 P 384f5045b6a1446b8de95d923f9e10c2 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 384f5045b6a1446b8de95d923f9e10c2. Latest consensus state: current_term: 1 leader_uuid: "384f5045b6a1446b8de95d923f9e10c2" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "384f5045b6a1446b8de95d923f9e10c2" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 38829 } } }
I20260620 14:50:21.970940 12802 sys_catalog.cc:458] T 00000000000000000000000000000000 P 384f5045b6a1446b8de95d923f9e10c2 [sys.catalog]: This master's current role is: LEADER
I20260620 14:50:21.976804 12801 sys_catalog.cc:455] T 00000000000000000000000000000000 P 384f5045b6a1446b8de95d923f9e10c2 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "384f5045b6a1446b8de95d923f9e10c2" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "384f5045b6a1446b8de95d923f9e10c2" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 38829 } } }
I20260620 14:50:21.977157 12801 sys_catalog.cc:458] T 00000000000000000000000000000000 P 384f5045b6a1446b8de95d923f9e10c2 [sys.catalog]: This master's current role is: LEADER
I20260620 14:50:21.977726 12795 ranger_client.cc:318] Using new properties file: /tmp/dist-test-taskUci3DU/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781966829091300-388-0/minicluster-data/master-0/logs/kudu-ranger-subprocess-log4j2.properties
I20260620 14:50:21.980454 12808 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 14:50:21.991405 12808 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 14:50:22.002492 12808 catalog_manager.cc:1347] Generated new cluster ID: 6e907ff4a8b346f4948255e02feaadd6
I20260620 14:50:22.002647 12808 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 14:50:22.037035 12808 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 14:50:22.039103 12808 catalog_manager.cc:1504] Loading token signing keys...
I20260620 14:50:22.051584 12808 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 384f5045b6a1446b8de95d923f9e10c2: Generated new TSK 0
I20260620 14:50:22.052873 12808 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.0.97.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskUci3DU/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781966829091300-388-0/krb5kdc/kudu_127.0.97.1.keytab.
Entry for principal kudu/127.0.97.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskUci3DU/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781966829091300-388-0/krb5kdc/kudu_127.0.97.1.keytab.
I20260620 14:50:22.830061 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskUci3DU/build/asan/bin/kudu
/tmp/dist-test-taskUci3DU/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskUci3DU/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781966829091300-388-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskUci3DU/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781966829091300-388-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskUci3DU/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781966829091300-388-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskUci3DU/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781966829091300-388-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.0.97.20:51495/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.1:0
--local_ip_for_outbound_sockets=127.0.97.1
--webserver_interface=127.0.97.1
--webserver_port=0
--tserver_master_addrs=127.0.97.62:38829
--keytab_file=/tmp/dist-test-taskUci3DU/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781966829091300-388-0/krb5kdc/kudu_127.0.97.1.keytab
--principal=kudu/127.0.97.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:34389
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskUci3DU/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781966829091300-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskUci3DU/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781966829091300-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskUci3DU/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781966829091300-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Jun 20 14:50:23 dist-test-slave-q9j6 krb5kdc[12284](info): AS_REQ (1 etypes {17}) 127.0.0.1: ISSUE: authtime 1781967023, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
W20260620 14:50:23.259212 12840 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 14:50:23.259553 12840 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 14:50:23.259773 12840 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 14:50:23.269811 12840 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 14:50:23.269968 12840 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 14:50:23.270154 12840 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.1
I20260620 14:50:23.281620 12840 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:34389
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskUci3DU/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781966829091300-388-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskUci3DU/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781966829091300-388-0/minicluster-data/ts-0/wal
--ranger_kms_url=127.0.97.20:51495/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskUci3DU/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781966829091300-388-0/krb5kdc/kudu_127.0.97.1.keytab
--principal=kudu/127.0.97.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskUci3DU/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781966829091300-388-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.97.1
--webserver_port=0
--tserver_master_addrs=127.0.97.62:38829
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.1
--log_dir=/tmp/dist-test-taskUci3DU/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781966829091300-388-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 13:41:20 UTC on ccycloud.quasar-vdagoi.root.comops.site
build id 4457462
ASAN enabled
I20260620 14:50:23.283182 12840 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 14:50:23.284632 12840 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 14:50:23.301250 12850 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 14:50:23.301205 12851 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 14:50:23.305716 12853 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 14:50:23.306499 12840 server_base.cc:1034] running on GCE node
I20260620 14:50:23.307251 12840 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 14:50:23.308465 12840 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 14:50:23.309726 12840 hybrid_clock.cc:648] HybridClock initialized: now 1781967023309648 us; error 68 us; skew 500 ppm
I20260620 14:50:23.310132 12840 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 14:50:23 dist-test-slave-q9j6 krb5kdc[12284](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781967023, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 14:50:23.314706 12840 init.cc:377] Logged in from keytab as kudu/127.0.97.1@KRBTEST.COM (short username kudu)
I20260620 14:50:23.320063 12840 webserver.cc:460] Webserver started at http://127.0.97.1:34797/ using document root <none> and password file <none>
I20260620 14:50:23.321179 12840 fs_manager.cc:359] Metadata directory not provided
I20260620 14:50:23.321354 12840 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 14:50:23.321830 12840 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Jun 20 14:50:23 dist-test-slave-q9j6 krb5kdc[12284](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781967023, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.1@KRBTEST.COM for HTTP/127.0.97.20@KRBTEST.COM
I20260620 14:50:23.349723 12840 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskUci3DU/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781966829091300-388-0/minicluster-data/ts-0/data/instance:
uuid: "a71cb9da2bb048879d6dac6997bcf785"
format_stamp: "Formatted at 2026-06-20 14:50:23 on dist-test-slave-q9j6"
server_key: "9b976df2f80143b18fdc044d9b444d5a"
server_key_iv: "01f45c251b5906705b55090aca1cc41b"
server_key_version: "kuduclusterkey@0"
I20260620 14:50:23.350900 12840 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskUci3DU/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781966829091300-388-0/minicluster-data/ts-0/wal/instance:
uuid: "a71cb9da2bb048879d6dac6997bcf785"
format_stamp: "Formatted at 2026-06-20 14:50:23 on dist-test-slave-q9j6"
server_key: "9b976df2f80143b18fdc044d9b444d5a"
server_key_iv: "01f45c251b5906705b55090aca1cc41b"
server_key_version: "kuduclusterkey@0"
I20260620 14:50:23.358552 12840 fs_manager.cc:691] Time spent creating directory manager: real 0.007s user 0.008s sys 0.000s
I20260620 14:50:23.385076 12860 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 14:50:23.387176 12840 fs_manager.cc:725] Time spent opening block manager: real 0.011s user 0.001s sys 0.007s
I20260620 14:50:23.387486 12840 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskUci3DU/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781966829091300-388-0/minicluster-data/ts-0/data,/tmp/dist-test-taskUci3DU/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781966829091300-388-0/minicluster-data/ts-0/wal
uuid: "a71cb9da2bb048879d6dac6997bcf785"
format_stamp: "Formatted at 2026-06-20 14:50:23 on dist-test-slave-q9j6"
server_key: "9b976df2f80143b18fdc044d9b444d5a"
server_key_iv: "01f45c251b5906705b55090aca1cc41b"
server_key_version: "kuduclusterkey@0"
I20260620 14:50:23.387866 12840 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskUci3DU/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781966829091300-388-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskUci3DU/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781966829091300-388-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskUci3DU/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781966829091300-388-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 14:50:23.420001 12840 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 14:50:23.425102 12840 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 14:50:23.425535 12840 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 14:50:23.426640 12840 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 14:50:23.434689 12840 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 14:50:23.434823 12840 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 14:50:23.434960 12840 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 14:50:23.435048 12840 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 14:50:23.516254 12840 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.1:42323
I20260620 14:50:23.516268 12972 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.1:42323 every 8 connection(s)
I20260620 14:50:23.518203 12840 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskUci3DU/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781966829091300-388-0/minicluster-data/ts-0/data/info.pb
I20260620 14:50:23.525686 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskUci3DU/build/asan/bin/kudu as pid 12840
I20260620 14:50:23.525900 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskUci3DU/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781966829091300-388-0/minicluster-data/ts-0/wal/instance
I20260620 14:50:23.526612 388 external_mini_cluster.cc:1428] Setting key b1bd47d8d22b699ba5f62e67b16e6770
Jun 20 14:50:23 dist-test-slave-q9j6 krb5kdc[12284](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781967023, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.1@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260620 14:50:23.565826 12976 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:50:23.524788 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.1:36673 (local address 127.0.97.62:38829)
0620 14:50:23.538255 (+ 13467us) server_negotiation.cc:206] Beginning negotiation
0620 14:50:23.538265 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0620 14:50:23.542379 (+ 4114us) server_negotiation.cc:407] Connection header received
0620 14:50:23.542463 (+ 84us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:50:23.542471 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:50:23.542598 (+ 127us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:50:23.542708 (+ 110us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:50:23.545923 (+ 3215us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:50:23.547214 (+ 1291us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:50:23.550103 (+ 2889us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:50:23.550912 (+ 809us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:50:23.551043 (+ 131us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:50:23.557029 (+ 5986us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 14:50:23.557037 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 14:50:23.557052 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 14:50:23.557099 (+ 47us) server_negotiation.cc:922] calling sasl_server_start()
0620 14:50:23.560925 (+ 3826us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:50:23.562569 (+ 1644us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:50:23.562575 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:50:23.562578 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:50:23.562753 (+ 175us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:50:23.563432 (+ 679us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:50:23.563437 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:50:23.563441 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:50:23.563840 (+ 399us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 14:50:23.564018 (+ 178us) server_negotiation.cc:1035] Waiting for connection context
0620 14:50:23.565138 (+ 1120us) server_negotiation.cc:299] Negotiation successful
0620 14:50:23.565414 (+ 276us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":13277,"thread_start_us":13466,"threads_started":1}
I20260620 14:50:23.568682 12975 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:50:23.523948 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:38829 (local address 127.0.97.1:36673)
0620 14:50:23.539037 (+ 15089us) negotiation.cc:107] Waiting for socket to connect
0620 14:50:23.539104 (+ 67us) client_negotiation.cc:174] Beginning negotiation
0620 14:50:23.541940 (+ 2836us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 14:50:23.543966 (+ 2026us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 14:50:23.543980 (+ 14us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 14:50:23.544716 (+ 736us) client_negotiation.cc:189] Negotiated authn=SASL
0620 14:50:23.545669 (+ 953us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:50:23.545702 (+ 33us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:50:23.547456 (+ 1754us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:50:23.547464 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:50:23.549887 (+ 2423us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:50:23.549908 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:50:23.551436 (+ 1528us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:50:23.551445 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:50:23.551859 (+ 414us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:50:23.552786 (+ 927us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 14:50:23.552824 (+ 38us) client_negotiation.cc:656] calling sasl_client_start()
0620 14:50:23.556700 (+ 3876us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 14:50:23.561670 (+ 4970us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 14:50:23.561681 (+ 11us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 14:50:23.561703 (+ 22us) client_negotiation.cc:760] calling sasl_client_step()
0620 14:50:23.562398 (+ 695us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 14:50:23.562920 (+ 522us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 14:50:23.562925 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 14:50:23.562928 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0620 14:50:23.563279 (+ 351us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 14:50:23.564182 (+ 903us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 14:50:23.564191 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 14:50:23.564653 (+ 462us) client_negotiation.cc:769] Sending connection context
0620 14:50:23.566400 (+ 1747us) client_negotiation.cc:240] Negotiation successful
0620 14:50:23.567036 (+ 636us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":14668,"thread_start_us":14900,"threads_started":1}
I20260620 14:50:23.570972 12973 heartbeater.cc:347] Connected to a master server at 127.0.97.62:38829
I20260620 14:50:23.571372 12973 heartbeater.cc:464] Registering TS with master...
I20260620 14:50:23.572332 12973 heartbeater.cc:511] Master 127.0.97.62:38829 requested a full tablet report, sending...
WARNING: no policy specified for kudu/127.0.97.2@KRBTEST.COM; defaulting to no policy
I20260620 14:50:23.575232 12760 ts_manager.cc:194] Registered new tserver with Master: a71cb9da2bb048879d6dac6997bcf785 (127.0.97.1:42323)
I20260620 14:50:23.576900 12760 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.97.1@KRBTEST.COM'} at 127.0.97.1:36673
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.2@KRBTEST.COM" created.
Jun 20 14:50:23 dist-test-slave-q9j6 krb5kdc[12284](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781967023, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.62@KRBTEST.COM for HTTP/127.0.97.20@KRBTEST.COM
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskUci3DU/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781966829091300-388-0/krb5kdc/kudu_127.0.97.2.keytab.
Entry for principal kudu/127.0.97.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskUci3DU/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781966829091300-388-0/krb5kdc/kudu_127.0.97.2.keytab.
I20260620 14:50:23.627530 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskUci3DU/build/asan/bin/kudu
/tmp/dist-test-taskUci3DU/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskUci3DU/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781966829091300-388-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskUci3DU/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781966829091300-388-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskUci3DU/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781966829091300-388-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskUci3DU/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781966829091300-388-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.0.97.20:51495/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.2:0
--local_ip_for_outbound_sockets=127.0.97.2
--webserver_interface=127.0.97.2
--webserver_port=0
--tserver_master_addrs=127.0.97.62:38829
--keytab_file=/tmp/dist-test-taskUci3DU/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781966829091300-388-0/krb5kdc/kudu_127.0.97.2.keytab
--principal=kudu/127.0.97.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:34389
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskUci3DU/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781966829091300-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskUci3DU/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781966829091300-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskUci3DU/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781966829091300-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Jun 20 14:50:23 dist-test-slave-q9j6 krb5kdc[12284](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781967023, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.62@KRBTEST.COM for HTTP/127.0.97.20@KRBTEST.COM
W20260620 14:50:24.151002 12979 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 14:50:24.151502 12979 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 14:50:24.151722 12979 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 14:50:24.163288 12979 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 14:50:24.163530 12979 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 14:50:24.163777 12979 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.2
I20260620 14:50:24.179826 12979 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:34389
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskUci3DU/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781966829091300-388-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskUci3DU/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781966829091300-388-0/minicluster-data/ts-1/wal
--ranger_kms_url=127.0.97.20:51495/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskUci3DU/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781966829091300-388-0/krb5kdc/kudu_127.0.97.2.keytab
--principal=kudu/127.0.97.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskUci3DU/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781966829091300-388-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.97.2
--webserver_port=0
--tserver_master_addrs=127.0.97.62:38829
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.2
--log_dir=/tmp/dist-test-taskUci3DU/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781966829091300-388-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 13:41:20 UTC on ccycloud.quasar-vdagoi.root.comops.site
build id 4457462
ASAN enabled
I20260620 14:50:24.181777 12979 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 14:50:24.183499 12979 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 14:50:24.214293 12987 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 14:50:24.217312 12986 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 14:50:24.221190 12990 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 14:50:24.221812 12979 server_base.cc:1034] running on GCE node
I20260620 14:50:24.222797 12979 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 14:50:24.227720 12979 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 14:50:24.229177 12979 hybrid_clock.cc:648] HybridClock initialized: now 1781967024229110 us; error 72 us; skew 500 ppm
I20260620 14:50:24.229600 12979 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 14:50:24 dist-test-slave-q9j6 krb5kdc[12284](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781967024, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 14:50:24.242957 12979 init.cc:377] Logged in from keytab as kudu/127.0.97.2@KRBTEST.COM (short username kudu)
I20260620 14:50:24.260850 12979 webserver.cc:460] Webserver started at http://127.0.97.2:38799/ using document root <none> and password file <none>
I20260620 14:50:24.261929 12979 fs_manager.cc:359] Metadata directory not provided
I20260620 14:50:24.262107 12979 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 14:50:24.262547 12979 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Jun 20 14:50:24 dist-test-slave-q9j6 krb5kdc[12284](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781967024, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.2@KRBTEST.COM for HTTP/127.0.97.20@KRBTEST.COM
I20260620 14:50:24.286602 12979 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskUci3DU/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781966829091300-388-0/minicluster-data/ts-1/data/instance:
uuid: "4868180fab3242e691c013a76dd5cb47"
format_stamp: "Formatted at 2026-06-20 14:50:24 on dist-test-slave-q9j6"
server_key: "c1f45804186a421654d6c4522fa51496"
server_key_iv: "10dfdcdac2b99de72f26f4f1cbc62b2a"
server_key_version: "kuduclusterkey@0"
I20260620 14:50:24.287819 12979 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskUci3DU/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781966829091300-388-0/minicluster-data/ts-1/wal/instance:
uuid: "4868180fab3242e691c013a76dd5cb47"
format_stamp: "Formatted at 2026-06-20 14:50:24 on dist-test-slave-q9j6"
server_key: "c1f45804186a421654d6c4522fa51496"
server_key_iv: "10dfdcdac2b99de72f26f4f1cbc62b2a"
server_key_version: "kuduclusterkey@0"
I20260620 14:50:24.295930 12979 fs_manager.cc:691] Time spent creating directory manager: real 0.008s user 0.007s sys 0.000s
I20260620 14:50:24.326761 13003 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 14:50:24.329082 12979 fs_manager.cc:725] Time spent opening block manager: real 0.011s user 0.005s sys 0.004s
I20260620 14:50:24.329399 12979 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskUci3DU/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781966829091300-388-0/minicluster-data/ts-1/data,/tmp/dist-test-taskUci3DU/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781966829091300-388-0/minicluster-data/ts-1/wal
uuid: "4868180fab3242e691c013a76dd5cb47"
format_stamp: "Formatted at 2026-06-20 14:50:24 on dist-test-slave-q9j6"
server_key: "c1f45804186a421654d6c4522fa51496"
server_key_iv: "10dfdcdac2b99de72f26f4f1cbc62b2a"
server_key_version: "kuduclusterkey@0"
I20260620 14:50:24.329768 12979 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskUci3DU/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781966829091300-388-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskUci3DU/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781966829091300-388-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskUci3DU/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781966829091300-388-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 14:50:24.367055 12979 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 14:50:24.372607 12979 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 14:50:24.372949 12979 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 14:50:24.374280 12979 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 14:50:24.376111 12979 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 14:50:24.376204 12979 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 14:50:24.376377 12979 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 14:50:24.376441 12979 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 14:50:24.417089 12979 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.2:34423
I20260620 14:50:24.417129 13115 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.2:34423 every 8 connection(s)
I20260620 14:50:24.418876 12979 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskUci3DU/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781966829091300-388-0/minicluster-data/ts-1/data/info.pb
I20260620 14:50:24.425941 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskUci3DU/build/asan/bin/kudu as pid 12979
I20260620 14:50:24.426137 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskUci3DU/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781966829091300-388-0/minicluster-data/ts-1/wal/instance
I20260620 14:50:24.426656 388 external_mini_cluster.cc:1428] Setting key ebde722e3240683c7efcee78058f3ebc
Jun 20 14:50:24 dist-test-slave-q9j6 krb5kdc[12284](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781967024, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.2@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260620 14:50:24.452710 13118 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:50:24.422663 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:38829 (local address 127.0.97.2:42553)
0620 14:50:24.425066 (+ 2403us) negotiation.cc:107] Waiting for socket to connect
0620 14:50:24.425122 (+ 56us) client_negotiation.cc:174] Beginning negotiation
0620 14:50:24.427461 (+ 2339us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 14:50:24.432879 (+ 5418us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 14:50:24.432893 (+ 14us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 14:50:24.433542 (+ 649us) client_negotiation.cc:189] Negotiated authn=SASL
0620 14:50:24.434420 (+ 878us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:50:24.434447 (+ 27us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:50:24.436117 (+ 1670us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:50:24.436122 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:50:24.438183 (+ 2061us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:50:24.438206 (+ 23us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:50:24.439264 (+ 1058us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:50:24.439271 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:50:24.439713 (+ 442us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:50:24.440902 (+ 1189us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 14:50:24.440935 (+ 33us) client_negotiation.cc:656] calling sasl_client_start()
0620 14:50:24.443712 (+ 2777us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 14:50:24.447731 (+ 4019us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 14:50:24.447740 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 14:50:24.447759 (+ 19us) client_negotiation.cc:760] calling sasl_client_step()
0620 14:50:24.448372 (+ 613us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 14:50:24.448834 (+ 462us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 14:50:24.448839 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 14:50:24.448842 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0620 14:50:24.449100 (+ 258us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 14:50:24.449799 (+ 699us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 14:50:24.449807 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 14:50:24.450269 (+ 462us) client_negotiation.cc:769] Sending connection context
0620 14:50:24.450592 (+ 323us) client_negotiation.cc:240] Negotiation successful
0620 14:50:24.451163 (+ 571us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":2081,"thread_start_us":1859,"threads_started":1}
I20260620 14:50:24.454130 13119 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:50:24.423296 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.2:42553 (local address 127.0.97.62:38829)
0620 14:50:24.428514 (+ 5218us) server_negotiation.cc:206] Beginning negotiation
0620 14:50:24.428520 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 14:50:24.428546 (+ 26us) server_negotiation.cc:407] Connection header received
0620 14:50:24.432215 (+ 3669us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:50:24.432221 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:50:24.432311 (+ 90us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:50:24.432411 (+ 100us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:50:24.434697 (+ 2286us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:50:24.435916 (+ 1219us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:50:24.438379 (+ 2463us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:50:24.438977 (+ 598us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:50:24.439119 (+ 142us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:50:24.444023 (+ 4904us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 14:50:24.444030 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 14:50:24.444035 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 14:50:24.444076 (+ 41us) server_negotiation.cc:922] calling sasl_server_start()
0620 14:50:24.447505 (+ 3429us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:50:24.448527 (+ 1022us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:50:24.448532 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:50:24.448535 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:50:24.448699 (+ 164us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:50:24.449233 (+ 534us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:50:24.449238 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:50:24.449240 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:50:24.449586 (+ 346us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 14:50:24.449719 (+ 133us) server_negotiation.cc:1035] Waiting for connection context
0620 14:50:24.453303 (+ 3584us) server_negotiation.cc:299] Negotiation successful
0620 14:50:24.453602 (+ 299us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":5061,"thread_start_us":4908,"threads_started":1}
WARNING: no policy specified for kudu/127.0.97.3@KRBTEST.COM; defaulting to no policy
I20260620 14:50:24.456643 13116 heartbeater.cc:347] Connected to a master server at 127.0.97.62:38829
I20260620 14:50:24.456991 13116 heartbeater.cc:464] Registering TS with master...
I20260620 14:50:24.457860 13116 heartbeater.cc:511] Master 127.0.97.62:38829 requested a full tablet report, sending...
I20260620 14:50:24.459722 12760 ts_manager.cc:194] Registered new tserver with Master: 4868180fab3242e691c013a76dd5cb47 (127.0.97.2:34423)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.3@KRBTEST.COM" created.
I20260620 14:50:24.460819 12760 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.97.2@KRBTEST.COM'} at 127.0.97.2:42553
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskUci3DU/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781966829091300-388-0/krb5kdc/kudu_127.0.97.3.keytab.
Entry for principal kudu/127.0.97.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskUci3DU/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781966829091300-388-0/krb5kdc/kudu_127.0.97.3.keytab.
I20260620 14:50:24.494722 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskUci3DU/build/asan/bin/kudu
/tmp/dist-test-taskUci3DU/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskUci3DU/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781966829091300-388-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskUci3DU/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781966829091300-388-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskUci3DU/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781966829091300-388-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskUci3DU/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781966829091300-388-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.0.97.20:51495/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.3:0
--local_ip_for_outbound_sockets=127.0.97.3
--webserver_interface=127.0.97.3
--webserver_port=0
--tserver_master_addrs=127.0.97.62:38829
--keytab_file=/tmp/dist-test-taskUci3DU/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781966829091300-388-0/krb5kdc/kudu_127.0.97.3.keytab
--principal=kudu/127.0.97.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:34389
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskUci3DU/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781966829091300-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskUci3DU/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781966829091300-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskUci3DU/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781966829091300-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260620 14:50:24.580595 12973 heartbeater.cc:503] Master 127.0.97.62:38829 was elected leader, sending a full tablet report...
W20260620 14:50:24.891680 13122 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 14:50:24.892045 13122 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 14:50:24.892221 13122 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 14:50:24.900004 13122 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 14:50:24.900121 13122 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 14:50:24.900321 13122 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.3
I20260620 14:50:24.910367 13122 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:34389
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskUci3DU/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781966829091300-388-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskUci3DU/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781966829091300-388-0/minicluster-data/ts-2/wal
--ranger_kms_url=127.0.97.20:51495/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskUci3DU/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781966829091300-388-0/krb5kdc/kudu_127.0.97.3.keytab
--principal=kudu/127.0.97.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskUci3DU/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781966829091300-388-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.97.3
--webserver_port=0
--tserver_master_addrs=127.0.97.62:38829
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.3
--log_dir=/tmp/dist-test-taskUci3DU/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781966829091300-388-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 13:41:20 UTC on ccycloud.quasar-vdagoi.root.comops.site
build id 4457462
ASAN enabled
I20260620 14:50:24.911788 13122 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 14:50:24.913160 13122 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 14:50:24.927031 13128 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 14:50:24.927569 13127 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 14:50:24.927841 13130 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 14:50:24.929239 13122 server_base.cc:1034] running on GCE node
I20260620 14:50:24.929970 13122 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 14:50:24.930908 13122 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 14:50:24.932157 13122 hybrid_clock.cc:648] HybridClock initialized: now 1781967024932114 us; error 39 us; skew 500 ppm
I20260620 14:50:24.932533 13122 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 14:50:24 dist-test-slave-q9j6 krb5kdc[12284](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781967024, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 14:50:24.936558 13122 init.cc:377] Logged in from keytab as kudu/127.0.97.3@KRBTEST.COM (short username kudu)
I20260620 14:50:24.938125 13122 webserver.cc:460] Webserver started at http://127.0.97.3:38267/ using document root <none> and password file <none>
I20260620 14:50:24.938902 13122 fs_manager.cc:359] Metadata directory not provided
I20260620 14:50:24.939047 13122 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 14:50:24.939385 13122 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Jun 20 14:50:24 dist-test-slave-q9j6 krb5kdc[12284](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781967024, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.3@KRBTEST.COM for HTTP/127.0.97.20@KRBTEST.COM
I20260620 14:50:24.953310 13122 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskUci3DU/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781966829091300-388-0/minicluster-data/ts-2/data/instance:
uuid: "98035054f3114f6e900361e476f372db"
format_stamp: "Formatted at 2026-06-20 14:50:24 on dist-test-slave-q9j6"
server_key: "2d465eb908eae40e49fad2373cd7bb97"
server_key_iv: "2b55a16c024473e654d304e0eaf66a96"
server_key_version: "kuduclusterkey@0"
I20260620 14:50:24.954161 13122 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskUci3DU/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781966829091300-388-0/minicluster-data/ts-2/wal/instance:
uuid: "98035054f3114f6e900361e476f372db"
format_stamp: "Formatted at 2026-06-20 14:50:24 on dist-test-slave-q9j6"
server_key: "2d465eb908eae40e49fad2373cd7bb97"
server_key_iv: "2b55a16c024473e654d304e0eaf66a96"
server_key_version: "kuduclusterkey@0"
I20260620 14:50:24.959825 13122 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.003s sys 0.005s
I20260620 14:50:24.977432 13137 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 14:50:24.979259 13122 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.004s sys 0.000s
I20260620 14:50:24.979453 13122 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskUci3DU/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781966829091300-388-0/minicluster-data/ts-2/data,/tmp/dist-test-taskUci3DU/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781966829091300-388-0/minicluster-data/ts-2/wal
uuid: "98035054f3114f6e900361e476f372db"
format_stamp: "Formatted at 2026-06-20 14:50:24 on dist-test-slave-q9j6"
server_key: "2d465eb908eae40e49fad2373cd7bb97"
server_key_iv: "2b55a16c024473e654d304e0eaf66a96"
server_key_version: "kuduclusterkey@0"
I20260620 14:50:24.979743 13122 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskUci3DU/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781966829091300-388-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskUci3DU/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781966829091300-388-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskUci3DU/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781966829091300-388-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 14:50:25.009392 13122 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 14:50:25.014537 13122 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 14:50:25.014878 13122 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 14:50:25.016089 13122 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 14:50:25.017799 13122 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 14:50:25.017901 13122 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 14:50:25.018030 13122 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 14:50:25.018090 13122 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 14:50:25.052701 13122 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.3:44975
I20260620 14:50:25.052735 13249 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.3:44975 every 8 connection(s)
I20260620 14:50:25.054101 13122 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskUci3DU/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781966829091300-388-0/minicluster-data/ts-2/data/info.pb
I20260620 14:50:25.057664 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskUci3DU/build/asan/bin/kudu as pid 13122
I20260620 14:50:25.057869 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskUci3DU/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781966829091300-388-0/minicluster-data/ts-2/wal/instance
I20260620 14:50:25.058568 388 external_mini_cluster.cc:1428] Setting key 076c749322c0ce2463d0f81d16fd91bd
Jun 20 14:50:25 dist-test-slave-q9j6 krb5kdc[12284](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781967024, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.3@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260620 14:50:25.075444 13252 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:50:25.056916 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.3:46383 (local address 127.0.97.62:38829)
0620 14:50:25.057387 (+ 471us) server_negotiation.cc:206] Beginning negotiation
0620 14:50:25.057393 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 14:50:25.058747 (+ 1354us) server_negotiation.cc:407] Connection header received
0620 14:50:25.060916 (+ 2169us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:50:25.060922 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:50:25.061044 (+ 122us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:50:25.061164 (+ 120us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:50:25.062765 (+ 1601us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:50:25.063518 (+ 753us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:50:25.065259 (+ 1741us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:50:25.065715 (+ 456us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:50:25.065857 (+ 142us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:50:25.069353 (+ 3496us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 14:50:25.069358 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 14:50:25.069362 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 14:50:25.069396 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0620 14:50:25.071937 (+ 2541us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:50:25.072925 (+ 988us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:50:25.072930 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:50:25.072933 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:50:25.073123 (+ 190us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:50:25.073699 (+ 576us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:50:25.073704 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:50:25.073707 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:50:25.074063 (+ 356us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 14:50:25.074230 (+ 167us) server_negotiation.cc:1035] Waiting for connection context
0620 14:50:25.075013 (+ 783us) server_negotiation.cc:299] Negotiation successful
0620 14:50:25.075199 (+ 186us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":357,"thread_start_us":214,"threads_started":1}
I20260620 14:50:25.076520 13253 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:50:25.057633 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:38829 (local address 127.0.97.3:46383)
0620 14:50:25.058488 (+ 855us) negotiation.cc:107] Waiting for socket to connect
0620 14:50:25.058545 (+ 57us) client_negotiation.cc:174] Beginning negotiation
0620 14:50:25.060621 (+ 2076us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 14:50:25.061416 (+ 795us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 14:50:25.061425 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 14:50:25.061981 (+ 556us) client_negotiation.cc:189] Negotiated authn=SASL
0620 14:50:25.062597 (+ 616us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:50:25.062616 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:50:25.063719 (+ 1103us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:50:25.063723 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:50:25.065109 (+ 1386us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:50:25.065121 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:50:25.065947 (+ 826us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:50:25.065954 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:50:25.066208 (+ 254us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:50:25.066857 (+ 649us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 14:50:25.066881 (+ 24us) client_negotiation.cc:656] calling sasl_client_start()
0620 14:50:25.069052 (+ 2171us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 14:50:25.072115 (+ 3063us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 14:50:25.072124 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 14:50:25.072190 (+ 66us) client_negotiation.cc:760] calling sasl_client_step()
0620 14:50:25.072787 (+ 597us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 14:50:25.073257 (+ 470us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 14:50:25.073263 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 14:50:25.073269 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0620 14:50:25.073538 (+ 269us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 14:50:25.074219 (+ 681us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 14:50:25.074229 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 14:50:25.074655 (+ 426us) client_negotiation.cc:769] Sending connection context
0620 14:50:25.074889 (+ 234us) client_negotiation.cc:240] Negotiation successful
0620 14:50:25.075289 (+ 400us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":599,"thread_start_us":376,"threads_started":1}
I20260620 14:50:25.077979 13250 heartbeater.cc:347] Connected to a master server at 127.0.97.62:38829
I20260620 14:50:25.078320 13250 heartbeater.cc:464] Registering TS with master...
I20260620 14:50:25.079048 13250 heartbeater.cc:511] Master 127.0.97.62:38829 requested a full tablet report, sending...
I20260620 14:50:25.080579 12760 ts_manager.cc:194] Registered new tserver with Master: 98035054f3114f6e900361e476f372db (127.0.97.3:44975)
I20260620 14:50:25.081403 12760 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.97.3@KRBTEST.COM'} at 127.0.97.3:46383
I20260620 14:50:25.086786 388 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260620 14:50:25.088503 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskUci3DU/build/asan/bin/kudu pid 12840
Jun 20 14:50:25 dist-test-slave-q9j6 krb5kdc[12284](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781967021, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.1@KRBTEST.COM
I20260620 14:50:25.105902 13255 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:50:25.089399 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41086 (local address 127.0.97.1:42323)
0620 14:50:25.090039 (+ 640us) server_negotiation.cc:206] Beginning negotiation
0620 14:50:25.090046 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0620 14:50:25.090072 (+ 26us) server_negotiation.cc:407] Connection header received
0620 14:50:25.090133 (+ 61us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:50:25.090140 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:50:25.090344 (+ 204us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:50:25.090556 (+ 212us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:50:25.092428 (+ 1872us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:50:25.093466 (+ 1038us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:50:25.094955 (+ 1489us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:50:25.095534 (+ 579us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:50:25.095729 (+ 195us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:50:25.100090 (+ 4361us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 14:50:25.100105 (+ 15us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 14:50:25.100122 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 14:50:25.100166 (+ 44us) server_negotiation.cc:922] calling sasl_server_start()
0620 14:50:25.102871 (+ 2705us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:50:25.103657 (+ 786us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:50:25.103673 (+ 16us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:50:25.103675 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:50:25.103822 (+ 147us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:50:25.104229 (+ 407us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:50:25.104235 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:50:25.104238 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:50:25.104602 (+ 364us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 14:50:25.104933 (+ 331us) server_negotiation.cc:1035] Waiting for connection context
0620 14:50:25.105258 (+ 325us) server_negotiation.cc:299] Negotiation successful
0620 14:50:25.105555 (+ 297us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":485,"thread_start_us":283,"threads_started":1}
I20260620 14:50:25.107224 12968 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 14:50:25.192619 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskUci3DU/build/asan/bin/kudu with pid 12840
I20260620 14:50:25.208326 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskUci3DU/build/asan/bin/kudu pid 12979
Jun 20 14:50:25 dist-test-slave-q9j6 krb5kdc[12284](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781967021, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.2@KRBTEST.COM
I20260620 14:50:25.224781 13257 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:50:25.209254 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40444 (local address 127.0.97.2:34423)
0620 14:50:25.209860 (+ 606us) server_negotiation.cc:206] Beginning negotiation
0620 14:50:25.209868 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0620 14:50:25.209894 (+ 26us) server_negotiation.cc:407] Connection header received
0620 14:50:25.209958 (+ 64us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:50:25.209965 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:50:25.210194 (+ 229us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:50:25.210331 (+ 137us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:50:25.211691 (+ 1360us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:50:25.212691 (+ 1000us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:50:25.214054 (+ 1363us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:50:25.214614 (+ 560us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:50:25.214712 (+ 98us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:50:25.219114 (+ 4402us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 14:50:25.219127 (+ 13us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 14:50:25.219143 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 14:50:25.219181 (+ 38us) server_negotiation.cc:922] calling sasl_server_start()
0620 14:50:25.222037 (+ 2856us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:50:25.222736 (+ 699us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:50:25.222743 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:50:25.222745 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:50:25.222879 (+ 134us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:50:25.223191 (+ 312us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:50:25.223194 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:50:25.223196 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:50:25.223536 (+ 340us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 14:50:25.223859 (+ 323us) server_negotiation.cc:1035] Waiting for connection context
0620 14:50:25.224146 (+ 287us) server_negotiation.cc:299] Negotiation successful
0620 14:50:25.224357 (+ 211us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":458,"thread_start_us":343,"threads_started":1}
I20260620 14:50:25.226116 13111 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 14:50:25.308349 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskUci3DU/build/asan/bin/kudu with pid 12979
I20260620 14:50:25.324054 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskUci3DU/build/asan/bin/kudu pid 13122
Jun 20 14:50:25 dist-test-slave-q9j6 krb5kdc[12284](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781967021, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.3@KRBTEST.COM
I20260620 14:50:25.340370 13259 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:50:25.325074 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38592 (local address 127.0.97.3:44975)
0620 14:50:25.325686 (+ 612us) server_negotiation.cc:206] Beginning negotiation
0620 14:50:25.325693 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0620 14:50:25.325721 (+ 28us) server_negotiation.cc:407] Connection header received
0620 14:50:25.325779 (+ 58us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:50:25.325785 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:50:25.326015 (+ 230us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:50:25.326142 (+ 127us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:50:25.327455 (+ 1313us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:50:25.328523 (+ 1068us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:50:25.329750 (+ 1227us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:50:25.330318 (+ 568us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:50:25.330416 (+ 98us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:50:25.334624 (+ 4208us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 14:50:25.334636 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 14:50:25.334651 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 14:50:25.334687 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0620 14:50:25.337593 (+ 2906us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:50:25.338353 (+ 760us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:50:25.338359 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:50:25.338362 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:50:25.338489 (+ 127us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:50:25.338860 (+ 371us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:50:25.338863 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:50:25.338865 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:50:25.339211 (+ 346us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 14:50:25.339481 (+ 270us) server_negotiation.cc:1035] Waiting for connection context
0620 14:50:25.339773 (+ 292us) server_negotiation.cc:299] Negotiation successful
0620 14:50:25.339969 (+ 196us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":469,"thread_start_us":308,"threads_started":1}
I20260620 14:50:25.341673 13245 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 14:50:25.428953 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskUci3DU/build/asan/bin/kudu with pid 13122
I20260620 14:50:25.444862 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskUci3DU/build/asan/bin/kudu pid 12728
I20260620 14:50:25.446182 12790 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 14:50:25.554121 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskUci3DU/build/asan/bin/kudu with pid 12728
I20260620 14:50:25.576416 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskUci3DU/build/asan/bin/kudu
/tmp/dist-test-taskUci3DU/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskUci3DU/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781966829091300-388-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskUci3DU/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781966829091300-388-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskUci3DU/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781966829091300-388-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskUci3DU/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781966829091300-388-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:38829
--webserver_interface=127.0.97.62
--webserver_port=33863
--keytab_file=/tmp/dist-test-taskUci3DU/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781966829091300-388-0/krb5kdc/kudu_127.0.97.62.keytab
--principal=kudu/127.0.97.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:34389
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.97.62:38829
--ranger_config_path=/tmp/dist-test-taskUci3DU/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781966829091300-388-0/minicluster-data/ranger-client
--trusted_user_acl=test-admin
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.0.97.20:51495/kms
--rpc_trace_negotiation
--ranger_kms_url=invalid.host:1234/kms,127.0.97.20:51495/kms with env {KRB5CCNAME=/tmp/dist-test-taskUci3DU/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781966829091300-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskUci3DU/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781966829091300-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskUci3DU/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781966829091300-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 14:50:25.913726 13262 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 14:50:25.914017 13262 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 14:50:25.914186 13262 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 14:50:25.921638 13262 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 14:50:25.921777 13262 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 14:50:25.921862 13262 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 14:50:25.921965 13262 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 14:50:25.922043 13262 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 14:50:25.932391 13262 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:34389
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskUci3DU/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781966829091300-388-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskUci3DU/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781966829091300-388-0/minicluster-data/master-0/wal
--ranger_kms_url=invalid.host:1234/kms,127.0.97.20:51495/kms
--trusted_user_acl=<redacted>
--ipki_ca_key_size=768
--master_addresses=127.0.97.62:38829
--ranger_config_path=/tmp/dist-test-taskUci3DU/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781966829091300-388-0/minicluster-data/ranger-client
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskUci3DU/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781966829091300-388-0/krb5kdc/kudu_127.0.97.62.keytab
--principal=kudu/127.0.97.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:38829
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskUci3DU/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781966829091300-388-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.97.62
--webserver_port=33863
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskUci3DU/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781966829091300-388-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 13:41:20 UTC on ccycloud.quasar-vdagoi.root.comops.site
build id 4457462
ASAN enabled
I20260620 14:50:25.933794 13262 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 14:50:25.935206 13262 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 14:50:25.948009 13268 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 14:50:25.948040 13270 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 14:50:25.948163 13267 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 14:50:25.948676 13262 server_base.cc:1034] running on GCE node
I20260620 14:50:25.949580 13262 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 14:50:25.951260 13262 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 14:50:25.952522 13262 hybrid_clock.cc:648] HybridClock initialized: now 1781967025952442 us; error 83 us; skew 500 ppm
I20260620 14:50:25.952920 13262 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 14:50:25 dist-test-slave-q9j6 krb5kdc[12284](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781967025, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 14:50:25.956323 13262 init.cc:377] Logged in from keytab as kudu/127.0.97.62@KRBTEST.COM (short username kudu)
I20260620 14:50:25.958021 13262 webserver.cc:460] Webserver started at http://127.0.97.62:33863/ using document root <none> and password file <none>
I20260620 14:50:25.958832 13262 fs_manager.cc:359] Metadata directory not provided
I20260620 14:50:25.958994 13262 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
Jun 20 14:50:26 dist-test-slave-q9j6 krb5kdc[12284](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781967025, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.62@KRBTEST.COM for HTTP/127.0.97.20@KRBTEST.COM
I20260620 14:50:26.044253 13262 fs_manager.cc:709] Time spent opening directory manager: real 0.005s user 0.001s sys 0.004s
I20260620 14:50:26.048108 13278 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 14:50:26.050009 13262 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.004s sys 0.000s
I20260620 14:50:26.050243 13262 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskUci3DU/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781966829091300-388-0/minicluster-data/master-0/data,/tmp/dist-test-taskUci3DU/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781966829091300-388-0/minicluster-data/master-0/wal
uuid: "384f5045b6a1446b8de95d923f9e10c2"
format_stamp: "Formatted at 2026-06-20 14:50:21 on dist-test-slave-q9j6"
server_key: "d06409155333808df639af13e3ed2e42"
server_key_iv: "621d8741b12cd60fe34d912ab3cd141f"
server_key_version: "kuduclusterkey@0"
I20260620 14:50:26.051733 13262 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskUci3DU/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781966829091300-388-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskUci3DU/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781966829091300-388-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskUci3DU/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781966829091300-388-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 14:50:26.070766 13262 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 14:50:26.074793 13262 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 14:50:26.075062 13262 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 14:50:26.094558 13262 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.62:38829
I20260620 14:50:26.094579 13329 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.62:38829 every 8 connection(s)
I20260620 14:50:26.096381 13262 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskUci3DU/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781966829091300-388-0/minicluster-data/master-0/data/info.pb
I20260620 14:50:26.102034 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskUci3DU/build/asan/bin/kudu as pid 13262
I20260620 14:50:26.102857 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskUci3DU/build/asan/bin/kudu
/tmp/dist-test-taskUci3DU/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskUci3DU/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781966829091300-388-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskUci3DU/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781966829091300-388-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskUci3DU/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781966829091300-388-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskUci3DU/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781966829091300-388-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.0.97.20:51495/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.1:42323
--local_ip_for_outbound_sockets=127.0.97.1
--tserver_master_addrs=127.0.97.62:38829
--webserver_port=34797
--webserver_interface=127.0.97.1
--keytab_file=/tmp/dist-test-taskUci3DU/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781966829091300-388-0/krb5kdc/kudu_127.0.97.1.keytab
--principal=kudu/127.0.97.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:34389
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--ranger_kms_url=invalid.host:1234/kms,127.0.97.20:51495/kms with env {KRB5CCNAME=/tmp/dist-test-taskUci3DU/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781966829091300-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskUci3DU/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781966829091300-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskUci3DU/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781966829091300-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260620 14:50:26.103768 13330 sys_catalog.cc:263] Verifying existing consensus state
I20260620 14:50:26.107120 13330 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 384f5045b6a1446b8de95d923f9e10c2: Bootstrap starting.
I20260620 14:50:26.126753 13330 log.cc:824] T 00000000000000000000000000000000 P 384f5045b6a1446b8de95d923f9e10c2: Log is configured to *not* fsync() on all Append() calls
I20260620 14:50:26.136121 13330 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 384f5045b6a1446b8de95d923f9e10c2: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=3 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260620 14:50:26.136746 13330 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 384f5045b6a1446b8de95d923f9e10c2: Bootstrap complete.
I20260620 14:50:26.146493 13330 raft_consensus.cc:348] T 00000000000000000000000000000000 P 384f5045b6a1446b8de95d923f9e10c2 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "384f5045b6a1446b8de95d923f9e10c2" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 38829 } }
I20260620 14:50:26.147434 13330 raft_consensus.cc:729] T 00000000000000000000000000000000 P 384f5045b6a1446b8de95d923f9e10c2 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 384f5045b6a1446b8de95d923f9e10c2, State: Initialized, Role: FOLLOWER
I20260620 14:50:26.148257 13330 consensus_queue.cc:260] T 00000000000000000000000000000000 P 384f5045b6a1446b8de95d923f9e10c2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "384f5045b6a1446b8de95d923f9e10c2" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 38829 } }
I20260620 14:50:26.148502 13330 raft_consensus.cc:388] T 00000000000000000000000000000000 P 384f5045b6a1446b8de95d923f9e10c2 [term 1 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 14:50:26.148626 13330 raft_consensus.cc:482] T 00000000000000000000000000000000 P 384f5045b6a1446b8de95d923f9e10c2 [term 1 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 14:50:26.148813 13330 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 384f5045b6a1446b8de95d923f9e10c2 [term 1 FOLLOWER]: Advancing to term 2
I20260620 14:50:26.152875 13330 raft_consensus.cc:504] T 00000000000000000000000000000000 P 384f5045b6a1446b8de95d923f9e10c2 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "384f5045b6a1446b8de95d923f9e10c2" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 38829 } }
I20260620 14:50:26.153440 13330 leader_election.cc:302] T 00000000000000000000000000000000 P 384f5045b6a1446b8de95d923f9e10c2 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 384f5045b6a1446b8de95d923f9e10c2; no voters:
I20260620 14:50:26.154103 13330 leader_election.cc:288] T 00000000000000000000000000000000 P 384f5045b6a1446b8de95d923f9e10c2 [CANDIDATE]: Term 2 election: Requested vote from peers
I20260620 14:50:26.154274 13334 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 384f5045b6a1446b8de95d923f9e10c2 [term 2 FOLLOWER]: Leader election won for term 2
I20260620 14:50:26.155465 13334 raft_consensus.cc:686] T 00000000000000000000000000000000 P 384f5045b6a1446b8de95d923f9e10c2 [term 2 LEADER]: Becoming Leader. State: Replica: 384f5045b6a1446b8de95d923f9e10c2, State: Running, Role: LEADER
I20260620 14:50:26.156188 13334 consensus_queue.cc:237] T 00000000000000000000000000000000 P 384f5045b6a1446b8de95d923f9e10c2 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 2, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "384f5045b6a1446b8de95d923f9e10c2" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 38829 } }
I20260620 14:50:26.156682 13330 sys_catalog.cc:564] T 00000000000000000000000000000000 P 384f5045b6a1446b8de95d923f9e10c2 [sys.catalog]: configured and running, proceeding with master startup.
I20260620 14:50:26.158918 13335 sys_catalog.cc:455] T 00000000000000000000000000000000 P 384f5045b6a1446b8de95d923f9e10c2 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 2 leader_uuid: "384f5045b6a1446b8de95d923f9e10c2" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "384f5045b6a1446b8de95d923f9e10c2" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 38829 } } }
I20260620 14:50:26.159381 13335 sys_catalog.cc:458] T 00000000000000000000000000000000 P 384f5045b6a1446b8de95d923f9e10c2 [sys.catalog]: This master's current role is: LEADER
I20260620 14:50:26.163317 13336 sys_catalog.cc:455] T 00000000000000000000000000000000 P 384f5045b6a1446b8de95d923f9e10c2 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 384f5045b6a1446b8de95d923f9e10c2. Latest consensus state: current_term: 2 leader_uuid: "384f5045b6a1446b8de95d923f9e10c2" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "384f5045b6a1446b8de95d923f9e10c2" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 38829 } } }
I20260620 14:50:26.163841 13336 sys_catalog.cc:458] T 00000000000000000000000000000000 P 384f5045b6a1446b8de95d923f9e10c2 [sys.catalog]: This master's current role is: LEADER
I20260620 14:50:26.165727 13340 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 14:50:26.173005 13340 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 14:50:26.175397 13340 catalog_manager.cc:1259] Loaded cluster ID: 6e907ff4a8b346f4948255e02feaadd6
I20260620 14:50:26.175639 13340 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 14:50:26.181164 13340 catalog_manager.cc:1504] Loading token signing keys...
I20260620 14:50:26.183518 13340 catalog_manager.cc:5879] T 00000000000000000000000000000000 P 384f5045b6a1446b8de95d923f9e10c2: Loaded TSK: 0
I20260620 14:50:26.185725 13340 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260620 14:50:26.188781 13330 ranger_client.cc:318] Using overwritten properties file: /tmp/dist-test-taskUci3DU/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781966829091300-388-0/minicluster-data/master-0/logs/kudu-ranger-subprocess-log4j2.properties
W20260620 14:50:26.596946 13332 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 14:50:26.597474 13332 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 14:50:26.597707 13332 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 14:50:26.610105 13332 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 14:50:26.610333 13332 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 14:50:26.610577 13332 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.1
I20260620 14:50:26.627363 13332 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:34389
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskUci3DU/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781966829091300-388-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskUci3DU/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781966829091300-388-0/minicluster-data/ts-0/wal
--ranger_kms_url=invalid.host:1234/kms,127.0.97.20:51495/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskUci3DU/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781966829091300-388-0/krb5kdc/kudu_127.0.97.1.keytab
--principal=kudu/127.0.97.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.1:42323
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskUci3DU/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781966829091300-388-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.97.1
--webserver_port=34797
--tserver_master_addrs=127.0.97.62:38829
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.1
--log_dir=/tmp/dist-test-taskUci3DU/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781966829091300-388-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 13:41:20 UTC on ccycloud.quasar-vdagoi.root.comops.site
build id 4457462
ASAN enabled
I20260620 14:50:26.629607 13332 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 14:50:26.631645 13332 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 14:50:26.670819 13367 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 14:50:26.672700 13368 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 14:50:26.681514 13332 server_base.cc:1034] running on GCE node
W20260620 14:50:26.681993 13370 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 14:50:26.682933 13332 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 14:50:26.684422 13332 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 14:50:26.688048 13332 hybrid_clock.cc:648] HybridClock initialized: now 1781967026687880 us; error 161 us; skew 500 ppm
I20260620 14:50:26.688609 13332 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 14:50:26 dist-test-slave-q9j6 krb5kdc[12284](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781967026, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 14:50:26.694005 13332 init.cc:377] Logged in from keytab as kudu/127.0.97.1@KRBTEST.COM (short username kudu)
I20260620 14:50:26.723327 13332 webserver.cc:460] Webserver started at http://127.0.97.1:34797/ using document root <none> and password file <none>
I20260620 14:50:26.724496 13332 fs_manager.cc:359] Metadata directory not provided
I20260620 14:50:26.724680 13332 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
Jun 20 14:50:26 dist-test-slave-q9j6 krb5kdc[12284](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781967026, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.1@KRBTEST.COM for HTTP/127.0.97.20@KRBTEST.COM
I20260620 14:50:26.760721 13332 fs_manager.cc:709] Time spent opening directory manager: real 0.007s user 0.005s sys 0.000s
I20260620 14:50:26.771411 13379 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 14:50:26.773612 13332 fs_manager.cc:725] Time spent opening block manager: real 0.011s user 0.005s sys 0.005s
I20260620 14:50:26.773890 13332 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskUci3DU/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781966829091300-388-0/minicluster-data/ts-0/data,/tmp/dist-test-taskUci3DU/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781966829091300-388-0/minicluster-data/ts-0/wal
uuid: "a71cb9da2bb048879d6dac6997bcf785"
format_stamp: "Formatted at 2026-06-20 14:50:23 on dist-test-slave-q9j6"
server_key: "9b976df2f80143b18fdc044d9b444d5a"
server_key_iv: "01f45c251b5906705b55090aca1cc41b"
server_key_version: "kuduclusterkey@0"
I20260620 14:50:26.775461 13332 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskUci3DU/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781966829091300-388-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskUci3DU/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781966829091300-388-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskUci3DU/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781966829091300-388-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 14:50:26.805894 13332 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 14:50:26.811596 13332 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 14:50:26.811973 13332 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 14:50:26.813294 13332 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 14:50:26.815133 13332 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 14:50:26.815248 13332 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 14:50:26.815351 13332 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 14:50:26.815419 13332 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 14:50:26.878965 13332 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.1:42323
I20260620 14:50:26.878983 13491 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.1:42323 every 8 connection(s)
I20260620 14:50:26.880998 13332 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskUci3DU/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781966829091300-388-0/minicluster-data/ts-0/data/info.pb
I20260620 14:50:26.889163 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskUci3DU/build/asan/bin/kudu as pid 13332
I20260620 14:50:26.890008 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskUci3DU/build/asan/bin/kudu
/tmp/dist-test-taskUci3DU/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskUci3DU/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781966829091300-388-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskUci3DU/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781966829091300-388-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskUci3DU/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781966829091300-388-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskUci3DU/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781966829091300-388-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.0.97.20:51495/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.2:34423
--local_ip_for_outbound_sockets=127.0.97.2
--tserver_master_addrs=127.0.97.62:38829
--webserver_port=38799
--webserver_interface=127.0.97.2
--keytab_file=/tmp/dist-test-taskUci3DU/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781966829091300-388-0/krb5kdc/kudu_127.0.97.2.keytab
--principal=kudu/127.0.97.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:34389
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--ranger_kms_url=invalid.host:1234/kms,127.0.97.20:51495/kms with env {KRB5CCNAME=/tmp/dist-test-taskUci3DU/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781966829091300-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskUci3DU/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781966829091300-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskUci3DU/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781966829091300-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Jun 20 14:50:26 dist-test-slave-q9j6 krb5kdc[12284](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781967026, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.1@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260620 14:50:26.916622 13494 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:50:26.884461 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:38829 (local address 127.0.97.1:39669)
0620 14:50:26.885372 (+ 911us) negotiation.cc:107] Waiting for socket to connect
0620 14:50:26.885431 (+ 59us) client_negotiation.cc:174] Beginning negotiation
0620 14:50:26.887573 (+ 2142us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 14:50:26.889730 (+ 2157us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 14:50:26.889742 (+ 12us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 14:50:26.890298 (+ 556us) client_negotiation.cc:189] Negotiated authn=SASL
0620 14:50:26.890921 (+ 623us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:50:26.890942 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:50:26.893346 (+ 2404us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:50:26.893353 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:50:26.895663 (+ 2310us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:50:26.895687 (+ 24us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:50:26.897633 (+ 1946us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:50:26.897638 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:50:26.897921 (+ 283us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:50:26.898826 (+ 905us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 14:50:26.898859 (+ 33us) client_negotiation.cc:656] calling sasl_client_start()
0620 14:50:26.905316 (+ 6457us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 14:50:26.910419 (+ 5103us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 14:50:26.910431 (+ 12us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 14:50:26.910450 (+ 19us) client_negotiation.cc:760] calling sasl_client_step()
0620 14:50:26.911191 (+ 741us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 14:50:26.911964 (+ 773us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 14:50:26.911970 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 14:50:26.911974 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0620 14:50:26.912280 (+ 306us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 14:50:26.913580 (+ 1300us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 14:50:26.913589 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 14:50:26.914092 (+ 503us) client_negotiation.cc:769] Sending connection context
0620 14:50:26.914447 (+ 355us) client_negotiation.cc:240] Negotiation successful
0620 14:50:26.915075 (+ 628us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":578,"thread_start_us":683,"threads_started":1}
I20260620 14:50:26.918051 13495 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:50:26.885802 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.1:39669 (local address 127.0.97.62:38829)
0620 14:50:26.886686 (+ 884us) server_negotiation.cc:206] Beginning negotiation
0620 14:50:26.886710 (+ 24us) server_negotiation.cc:399] Waiting for connection header
0620 14:50:26.886768 (+ 58us) server_negotiation.cc:407] Connection header received
0620 14:50:26.888303 (+ 1535us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:50:26.888318 (+ 15us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:50:26.888930 (+ 612us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:50:26.889546 (+ 616us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:50:26.891165 (+ 1619us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:50:26.893074 (+ 1909us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:50:26.895908 (+ 2834us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:50:26.897057 (+ 1149us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:50:26.897285 (+ 228us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:50:26.905752 (+ 8467us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 14:50:26.905765 (+ 13us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 14:50:26.905792 (+ 27us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 14:50:26.905856 (+ 64us) server_negotiation.cc:922] calling sasl_server_start()
0620 14:50:26.910142 (+ 4286us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:50:26.911427 (+ 1285us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:50:26.911438 (+ 11us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:50:26.911449 (+ 11us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:50:26.911779 (+ 330us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:50:26.912441 (+ 662us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:50:26.912447 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:50:26.912450 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:50:26.913226 (+ 776us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 14:50:26.913469 (+ 243us) server_negotiation.cc:1035] Waiting for connection context
0620 14:50:26.916043 (+ 2574us) server_negotiation.cc:299] Negotiation successful
0620 14:50:26.916568 (+ 525us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":608,"thread_start_us":336,"threads_started":1}
I20260620 14:50:26.922320 13492 heartbeater.cc:347] Connected to a master server at 127.0.97.62:38829
I20260620 14:50:26.922712 13492 heartbeater.cc:464] Registering TS with master...
I20260620 14:50:26.923754 13492 heartbeater.cc:511] Master 127.0.97.62:38829 requested a full tablet report, sending...
W20260620 14:50:26.927419 13492 heartbeater.cc:644] Failed to heartbeat to 127.0.97.62:38829 (0 consecutive failures): Service unavailable: Catalog manager is not initialized. State: Starting
I20260620 14:50:26.927716 13492 heartbeater.cc:464] Registering TS with master...
I20260620 14:50:26.928025 13492 heartbeater.cc:511] Master 127.0.97.62:38829 requested a full tablet report, sending...
I20260620 14:50:26.929332 13492 heartbeater.cc:464] Registering TS with master...
I20260620 14:50:26.929705 13492 heartbeater.cc:511] Master 127.0.97.62:38829 requested a full tablet report, sending...
W20260620 14:50:26.930905 13492 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260620 14:50:27.296557 13496 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 14:50:27.296928 13496 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 14:50:27.297101 13496 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 14:50:27.306224 13496 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 14:50:27.306394 13496 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 14:50:27.306576 13496 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.2
I20260620 14:50:27.317445 13496 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:34389
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskUci3DU/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781966829091300-388-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskUci3DU/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781966829091300-388-0/minicluster-data/ts-1/wal
--ranger_kms_url=invalid.host:1234/kms,127.0.97.20:51495/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskUci3DU/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781966829091300-388-0/krb5kdc/kudu_127.0.97.2.keytab
--principal=kudu/127.0.97.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.2:34423
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskUci3DU/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781966829091300-388-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.97.2
--webserver_port=38799
--tserver_master_addrs=127.0.97.62:38829
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.2
--log_dir=/tmp/dist-test-taskUci3DU/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781966829091300-388-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 13:41:20 UTC on ccycloud.quasar-vdagoi.root.comops.site
build id 4457462
ASAN enabled
I20260620 14:50:27.318862 13496 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 14:50:27.320272 13496 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 14:50:27.336333 13513 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 14:50:27.336580 13510 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 14:50:27.337271 13511 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 14:50:27.337500 13496 server_base.cc:1034] running on GCE node
I20260620 14:50:27.338394 13496 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 14:50:27.339705 13496 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 14:50:27.341039 13496 hybrid_clock.cc:648] HybridClock initialized: now 1781967027340961 us; error 62 us; skew 500 ppm
I20260620 14:50:27.341493 13496 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 14:50:27 dist-test-slave-q9j6 krb5kdc[12284](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781967027, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 14:50:27.348423 13496 init.cc:377] Logged in from keytab as kudu/127.0.97.2@KRBTEST.COM (short username kudu)
I20260620 14:50:27.373593 13496 webserver.cc:460] Webserver started at http://127.0.97.2:38799/ using document root <none> and password file <none>
I20260620 14:50:27.374609 13496 fs_manager.cc:359] Metadata directory not provided
I20260620 14:50:27.374787 13496 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
Jun 20 14:50:27 dist-test-slave-q9j6 krb5kdc[12284](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781967027, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.2@KRBTEST.COM for HTTP/127.0.97.20@KRBTEST.COM
I20260620 14:50:27.411484 13496 fs_manager.cc:709] Time spent opening directory manager: real 0.005s user 0.004s sys 0.000s
I20260620 14:50:27.414816 13523 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 14:50:27.416502 13496 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.000s sys 0.002s
I20260620 14:50:27.416713 13496 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskUci3DU/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781966829091300-388-0/minicluster-data/ts-1/data,/tmp/dist-test-taskUci3DU/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781966829091300-388-0/minicluster-data/ts-1/wal
uuid: "4868180fab3242e691c013a76dd5cb47"
format_stamp: "Formatted at 2026-06-20 14:50:24 on dist-test-slave-q9j6"
server_key: "c1f45804186a421654d6c4522fa51496"
server_key_iv: "10dfdcdac2b99de72f26f4f1cbc62b2a"
server_key_version: "kuduclusterkey@0"
I20260620 14:50:27.417861 13496 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskUci3DU/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781966829091300-388-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskUci3DU/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781966829091300-388-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskUci3DU/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781966829091300-388-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 14:50:27.435740 13496 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 14:50:27.439997 13496 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 14:50:27.440294 13496 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 14:50:27.441640 13496 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 14:50:27.443198 13496 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 14:50:27.443310 13496 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 14:50:27.443440 13496 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 14:50:27.443529 13496 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Jun 20 14:50:27 dist-test-slave-q9j6 krb5kdc[12284](info): AS_REQ (1 etypes {17}) 127.0.0.1: ISSUE: authtime 1781967027, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 14:50:27.507977 13496 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.2:34423
I20260620 14:50:27.508035 13635 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.2:34423 every 8 connection(s)
I20260620 14:50:27.510048 13496 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskUci3DU/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781966829091300-388-0/minicluster-data/ts-1/data/info.pb
I20260620 14:50:27.514832 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskUci3DU/build/asan/bin/kudu as pid 13496
I20260620 14:50:27.515873 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskUci3DU/build/asan/bin/kudu
/tmp/dist-test-taskUci3DU/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskUci3DU/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781966829091300-388-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskUci3DU/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781966829091300-388-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskUci3DU/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781966829091300-388-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskUci3DU/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781966829091300-388-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.0.97.20:51495/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.3:44975
--local_ip_for_outbound_sockets=127.0.97.3
--tserver_master_addrs=127.0.97.62:38829
--webserver_port=38267
--webserver_interface=127.0.97.3
--keytab_file=/tmp/dist-test-taskUci3DU/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781966829091300-388-0/krb5kdc/kudu_127.0.97.3.keytab
--principal=kudu/127.0.97.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:34389
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--ranger_kms_url=invalid.host:1234/kms,127.0.97.20:51495/kms with env {KRB5CCNAME=/tmp/dist-test-taskUci3DU/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781966829091300-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskUci3DU/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781966829091300-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskUci3DU/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781966829091300-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Jun 20 14:50:27 dist-test-slave-q9j6 krb5kdc[12284](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781967027, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.2@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260620 14:50:27.556149 13639 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:50:27.520003 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.2:38473 (local address 127.0.97.62:38829)
0620 14:50:27.536041 (+ 16038us) server_negotiation.cc:206] Beginning negotiation
0620 14:50:27.536048 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0620 14:50:27.536074 (+ 26us) server_negotiation.cc:407] Connection header received
0620 14:50:27.536143 (+ 69us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:50:27.536149 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:50:27.536242 (+ 93us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:50:27.536374 (+ 132us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:50:27.538626 (+ 2252us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:50:27.539836 (+ 1210us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:50:27.542427 (+ 2591us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:50:27.543076 (+ 649us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:50:27.543214 (+ 138us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:50:27.547797 (+ 4583us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 14:50:27.547805 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 14:50:27.547810 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 14:50:27.547857 (+ 47us) server_negotiation.cc:922] calling sasl_server_start()
0620 14:50:27.551452 (+ 3595us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:50:27.552727 (+ 1275us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:50:27.552737 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:50:27.552743 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:50:27.552942 (+ 199us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:50:27.553630 (+ 688us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:50:27.553637 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:50:27.553644 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:50:27.554099 (+ 455us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 14:50:27.554274 (+ 175us) server_negotiation.cc:1035] Waiting for connection context
0620 14:50:27.555475 (+ 1201us) server_negotiation.cc:299] Negotiation successful
0620 14:50:27.555771 (+ 296us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":15892,"thread_start_us":15782,"threads_started":1}
I20260620 14:50:27.557924 13638 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:50:27.519321 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:38829 (local address 127.0.97.2:38473)
0620 14:50:27.532307 (+ 12986us) negotiation.cc:107] Waiting for socket to connect
0620 14:50:27.532369 (+ 62us) client_negotiation.cc:174] Beginning negotiation
0620 14:50:27.535730 (+ 3361us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 14:50:27.536737 (+ 1007us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 14:50:27.536752 (+ 15us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 14:50:27.537476 (+ 724us) client_negotiation.cc:189] Negotiated authn=SASL
0620 14:50:27.538382 (+ 906us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:50:27.538412 (+ 30us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:50:27.540069 (+ 1657us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:50:27.540075 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:50:27.542184 (+ 2109us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:50:27.542228 (+ 44us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:50:27.543524 (+ 1296us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:50:27.543532 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:50:27.543868 (+ 336us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:50:27.544726 (+ 858us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 14:50:27.544759 (+ 33us) client_negotiation.cc:656] calling sasl_client_start()
0620 14:50:27.547442 (+ 2683us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 14:50:27.551681 (+ 4239us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 14:50:27.551695 (+ 14us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 14:50:27.551719 (+ 24us) client_negotiation.cc:760] calling sasl_client_step()
0620 14:50:27.552525 (+ 806us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 14:50:27.553125 (+ 600us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 14:50:27.553130 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 14:50:27.553137 (+ 7us) client_negotiation.cc:760] calling sasl_client_step()
0620 14:50:27.553464 (+ 327us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 14:50:27.554367 (+ 903us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 14:50:27.554378 (+ 11us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 14:50:27.554918 (+ 540us) client_negotiation.cc:769] Sending connection context
0620 14:50:27.555315 (+ 397us) client_negotiation.cc:240] Negotiation successful
0620 14:50:27.556024 (+ 709us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":12632,"thread_start_us":12375,"threads_started":1}
I20260620 14:50:27.559674 13636 heartbeater.cc:347] Connected to a master server at 127.0.97.62:38829
I20260620 14:50:27.560006 13636 heartbeater.cc:464] Registering TS with master...
I20260620 14:50:27.560883 13636 heartbeater.cc:511] Master 127.0.97.62:38829 requested a full tablet report, sending...
I20260620 14:50:27.563504 13295 ts_manager.cc:194] Registered new tserver with Master: 4868180fab3242e691c013a76dd5cb47 (127.0.97.2:34423)
I20260620 14:50:27.565804 13295 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.97.2@KRBTEST.COM'} at 127.0.97.2:38473
I20260620 14:50:27.931100 13492 heartbeater.cc:464] Registering TS with master...
I20260620 14:50:27.931561 13492 heartbeater.cc:511] Master 127.0.97.62:38829 requested a full tablet report, sending...
I20260620 14:50:27.933100 13295 ts_manager.cc:194] Registered new tserver with Master: a71cb9da2bb048879d6dac6997bcf785 (127.0.97.1:42323)
I20260620 14:50:27.934249 13295 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.97.1@KRBTEST.COM'} at 127.0.97.1:39669
Jun 20 14:50:27 dist-test-slave-q9j6 krb5kdc[12284](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781967027, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.62@KRBTEST.COM for HTTP/127.0.97.20@KRBTEST.COM
W20260620 14:50:27.966300 13640 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 14:50:27.966745 13640 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 14:50:27.966950 13640 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 14:50:27.978706 13640 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 14:50:27.978920 13640 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 14:50:27.979228 13640 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.3
I20260620 14:50:27.994715 13640 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:34389
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskUci3DU/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781966829091300-388-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskUci3DU/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781966829091300-388-0/minicluster-data/ts-2/wal
--ranger_kms_url=invalid.host:1234/kms,127.0.97.20:51495/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskUci3DU/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781966829091300-388-0/krb5kdc/kudu_127.0.97.3.keytab
--principal=kudu/127.0.97.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.3:44975
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskUci3DU/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781966829091300-388-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.97.3
--webserver_port=38267
--tserver_master_addrs=127.0.97.62:38829
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.3
--log_dir=/tmp/dist-test-taskUci3DU/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781966829091300-388-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 13:41:20 UTC on ccycloud.quasar-vdagoi.root.comops.site
build id 4457462
ASAN enabled
I20260620 14:50:27.996645 13640 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 14:50:27.998541 13640 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 14:50:28.035583 13646 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 14:50:28.047533 13647 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 14:50:28.062188 13640 server_base.cc:1034] running on GCE node
W20260620 14:50:28.062772 13650 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 14:50:28.063737 13640 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 14:50:28.071374 13640 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 14:50:28.073010 13640 hybrid_clock.cc:648] HybridClock initialized: now 1781967028072925 us; error 88 us; skew 500 ppm
I20260620 14:50:28.073536 13640 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 14:50:28 dist-test-slave-q9j6 krb5kdc[12284](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781967028, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 14:50:28.078632 13640 init.cc:377] Logged in from keytab as kudu/127.0.97.3@KRBTEST.COM (short username kudu)
I20260620 14:50:28.099701 13640 webserver.cc:460] Webserver started at http://127.0.97.3:38267/ using document root <none> and password file <none>
I20260620 14:50:28.100955 13640 fs_manager.cc:359] Metadata directory not provided
I20260620 14:50:28.101235 13640 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
Jun 20 14:50:28 dist-test-slave-q9j6 krb5kdc[12284](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781967028, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.3@KRBTEST.COM for HTTP/127.0.97.20@KRBTEST.COM
I20260620 14:50:28.138790 13640 fs_manager.cc:709] Time spent opening directory manager: real 0.006s user 0.005s sys 0.000s
I20260620 14:50:28.149117 13657 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 14:50:28.151247 13640 fs_manager.cc:725] Time spent opening block manager: real 0.010s user 0.005s sys 0.003s
I20260620 14:50:28.151533 13640 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskUci3DU/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781966829091300-388-0/minicluster-data/ts-2/data,/tmp/dist-test-taskUci3DU/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781966829091300-388-0/minicluster-data/ts-2/wal
uuid: "98035054f3114f6e900361e476f372db"
format_stamp: "Formatted at 2026-06-20 14:50:24 on dist-test-slave-q9j6"
server_key: "2d465eb908eae40e49fad2373cd7bb97"
server_key_iv: "2b55a16c024473e654d304e0eaf66a96"
server_key_version: "kuduclusterkey@0"
I20260620 14:50:28.153209 13640 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskUci3DU/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781966829091300-388-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskUci3DU/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781966829091300-388-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskUci3DU/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781966829091300-388-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 14:50:28.199528 13640 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 14:50:28.205097 13640 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 14:50:28.205472 13640 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 14:50:28.212980 13640 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 14:50:28.218374 13640 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 14:50:28.218588 13640 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 14:50:28.218752 13640 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 14:50:28.218863 13640 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Jun 20 14:50:28 dist-test-slave-q9j6 krb5kdc[12284](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781967027, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.62@KRBTEST.COM for HTTP/127.0.97.20@KRBTEST.COM
I20260620 14:50:28.485272 13640 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.3:44975
I20260620 14:50:28.485321 13770 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.3:44975 every 8 connection(s)
I20260620 14:50:28.487239 13640 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskUci3DU/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781966829091300-388-0/minicluster-data/ts-2/data/info.pb
I20260620 14:50:28.497169 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskUci3DU/build/asan/bin/kudu as pid 13640
Jun 20 14:50:28 dist-test-slave-q9j6 krb5kdc[12284](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781967028, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.3@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260620 14:50:28.538028 13774 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:50:28.498242 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40682 (local address 127.0.97.62:38829)
0620 14:50:28.514086 (+ 15844us) server_negotiation.cc:206] Beginning negotiation
0620 14:50:28.514096 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0620 14:50:28.514130 (+ 34us) server_negotiation.cc:407] Connection header received
0620 14:50:28.514229 (+ 99us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:50:28.514237 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:50:28.514346 (+ 109us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:50:28.514526 (+ 180us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:50:28.521457 (+ 6931us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:50:28.522790 (+ 1333us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:50:28.524999 (+ 2209us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:50:28.525683 (+ 684us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:50:28.525807 (+ 124us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:50:28.530140 (+ 4333us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 14:50:28.530148 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 14:50:28.530154 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 14:50:28.530199 (+ 45us) server_negotiation.cc:922] calling sasl_server_start()
0620 14:50:28.533817 (+ 3618us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:50:28.534852 (+ 1035us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:50:28.534858 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:50:28.534862 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:50:28.535028 (+ 166us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:50:28.535891 (+ 863us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:50:28.535898 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:50:28.535901 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:50:28.536304 (+ 403us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 14:50:28.536642 (+ 338us) server_negotiation.cc:1035] Waiting for connection context
0620 14:50:28.537142 (+ 500us) server_negotiation.cc:299] Negotiation successful
0620 14:50:28.537405 (+ 263us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":15632,"thread_start_us":12732,"threads_started":1}
I20260620 14:50:28.542061 13776 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:50:28.497812 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.3:56019 (local address 127.0.97.62:38829)
0620 14:50:28.508865 (+ 11053us) server_negotiation.cc:206] Beginning negotiation
0620 14:50:28.508872 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0620 14:50:28.519758 (+ 10886us) server_negotiation.cc:407] Connection header received
0620 14:50:28.519839 (+ 81us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:50:28.519844 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:50:28.519934 (+ 90us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:50:28.520052 (+ 118us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:50:28.522971 (+ 2919us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:50:28.524251 (+ 1280us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:50:28.526858 (+ 2607us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:50:28.527548 (+ 690us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:50:28.527717 (+ 169us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:50:28.534467 (+ 6750us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 14:50:28.534474 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 14:50:28.534478 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 14:50:28.534523 (+ 45us) server_negotiation.cc:922] calling sasl_server_start()
0620 14:50:28.538056 (+ 3533us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:50:28.539123 (+ 1067us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:50:28.539129 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:50:28.539133 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:50:28.539273 (+ 140us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:50:28.539897 (+ 624us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:50:28.539902 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:50:28.539905 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:50:28.540284 (+ 379us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 14:50:28.540416 (+ 132us) server_negotiation.cc:1035] Waiting for connection context
0620 14:50:28.541445 (+ 1029us) server_negotiation.cc:299] Negotiation successful
0620 14:50:28.541684 (+ 239us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":10907,"thread_start_us":15899,"threads_started":1}
I20260620 14:50:28.544528 13773 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:50:28.497104 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:38829 (local address 127.0.97.3:56019)
0620 14:50:28.517395 (+ 20291us) negotiation.cc:107] Waiting for socket to connect
0620 14:50:28.517461 (+ 66us) client_negotiation.cc:174] Beginning negotiation
0620 14:50:28.519325 (+ 1864us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 14:50:28.521078 (+ 1753us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 14:50:28.521092 (+ 14us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 14:50:28.521808 (+ 716us) client_negotiation.cc:189] Negotiated authn=SASL
0620 14:50:28.522699 (+ 891us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:50:28.522731 (+ 32us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:50:28.524456 (+ 1725us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:50:28.524462 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:50:28.526650 (+ 2188us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:50:28.526668 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:50:28.528061 (+ 1393us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:50:28.528069 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:50:28.528427 (+ 358us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:50:28.529573 (+ 1146us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 14:50:28.529612 (+ 39us) client_negotiation.cc:656] calling sasl_client_start()
0620 14:50:28.534144 (+ 4532us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 14:50:28.538238 (+ 4094us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 14:50:28.538249 (+ 11us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 14:50:28.538270 (+ 21us) client_negotiation.cc:760] calling sasl_client_step()
0620 14:50:28.538966 (+ 696us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 14:50:28.539412 (+ 446us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 14:50:28.539417 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 14:50:28.539421 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0620 14:50:28.539747 (+ 326us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 14:50:28.540511 (+ 764us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 14:50:28.540521 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 14:50:28.540988 (+ 467us) client_negotiation.cc:769] Sending connection context
0620 14:50:28.542337 (+ 1349us) client_negotiation.cc:240] Negotiation successful
0620 14:50:28.542966 (+ 629us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":19953,"thread_start_us":19694,"threads_started":1}
I20260620 14:50:28.546402 13771 heartbeater.cc:347] Connected to a master server at 127.0.97.62:38829
I20260620 14:50:28.546762 13771 heartbeater.cc:464] Registering TS with master...
I20260620 14:50:28.547740 13771 heartbeater.cc:511] Master 127.0.97.62:38829 requested a full tablet report, sending...
I20260620 14:50:28.549952 13295 ts_manager.cc:194] Registered new tserver with Master: 98035054f3114f6e900361e476f372db (127.0.97.3:44975)
I20260620 14:50:28.551121 13295 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.97.3@KRBTEST.COM'} at 127.0.97.3:56019
I20260620 14:50:28.553754 388 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260620 14:50:28.570957 13636 heartbeater.cc:503] Master 127.0.97.62:38829 was elected leader, sending a full tablet report...
I20260620 14:50:28.579993 13776 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:50:28.562479 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40698 (local address 127.0.97.62:38829)
0620 14:50:28.562676 (+ 197us) server_negotiation.cc:206] Beginning negotiation
0620 14:50:28.562683 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0620 14:50:28.564105 (+ 1422us) server_negotiation.cc:407] Connection header received
0620 14:50:28.564319 (+ 214us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:50:28.564325 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:50:28.564414 (+ 89us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:50:28.564532 (+ 118us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:50:28.566253 (+ 1721us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:50:28.567127 (+ 874us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:50:28.569159 (+ 2032us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:50:28.569850 (+ 691us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:50:28.569966 (+ 116us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:50:28.573762 (+ 3796us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 14:50:28.573770 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 14:50:28.573778 (+ 8us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 14:50:28.573827 (+ 49us) server_negotiation.cc:922] calling sasl_server_start()
0620 14:50:28.576577 (+ 2750us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:50:28.577362 (+ 785us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:50:28.577367 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:50:28.577369 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:50:28.577504 (+ 135us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:50:28.577916 (+ 412us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:50:28.577920 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:50:28.577922 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:50:28.578204 (+ 282us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 14:50:28.578454 (+ 250us) server_negotiation.cc:1035] Waiting for connection context
0620 14:50:28.578975 (+ 521us) server_negotiation.cc:299] Negotiation successful
0620 14:50:28.579495 (+ 520us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":69}
I20260620 14:50:28.585515 13295 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:40698:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260620 14:50:28.593146 13295 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260620 14:50:28.651212 13790 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:50:28.623823 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38594 (local address 127.0.97.3:44975)
0620 14:50:28.638506 (+ 14683us) server_negotiation.cc:206] Beginning negotiation
0620 14:50:28.638513 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0620 14:50:28.638584 (+ 71us) server_negotiation.cc:407] Connection header received
0620 14:50:28.638685 (+ 101us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:50:28.638695 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:50:28.638984 (+ 289us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:50:28.639195 (+ 211us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 14:50:28.640192 (+ 997us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:50:28.641972 (+ 1780us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:50:28.647997 (+ 6025us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:50:28.649505 (+ 1508us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:50:28.649659 (+ 154us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:50:28.649894 (+ 235us) server_negotiation.cc:1035] Waiting for connection context
0620 14:50:28.650689 (+ 795us) server_negotiation.cc:299] Negotiation successful
0620 14:50:28.650876 (+ 187us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":14514,"thread_start_us":14355,"threads_started":1}
I20260620 14:50:28.651455 13787 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:50:28.615929 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41100 (local address 127.0.97.1:42323)
0620 14:50:28.632296 (+ 16367us) server_negotiation.cc:206] Beginning negotiation
0620 14:50:28.632311 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0620 14:50:28.634343 (+ 2032us) server_negotiation.cc:407] Connection header received
0620 14:50:28.634489 (+ 146us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:50:28.634502 (+ 13us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:50:28.634888 (+ 386us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:50:28.635127 (+ 239us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 14:50:28.638057 (+ 2930us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:50:28.640747 (+ 2690us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:50:28.647309 (+ 6562us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:50:28.648898 (+ 1589us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:50:28.649031 (+ 133us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:50:28.649312 (+ 281us) server_negotiation.cc:1035] Waiting for connection context
0620 14:50:28.650944 (+ 1632us) server_negotiation.cc:299] Negotiation successful
0620 14:50:28.651115 (+ 171us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":16124,"thread_start_us":15938,"threads_started":1}
I20260620 14:50:28.652032 13788 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:50:28.619694 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:42323 (local address 127.0.0.1:41100)
0620 14:50:28.633711 (+ 14017us) negotiation.cc:107] Waiting for socket to connect
0620 14:50:28.633774 (+ 63us) client_negotiation.cc:174] Beginning negotiation
0620 14:50:28.634058 (+ 284us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 14:50:28.637099 (+ 3041us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 14:50:28.637111 (+ 12us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 14:50:28.637165 (+ 54us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 14:50:28.637732 (+ 567us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:50:28.637751 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:50:28.640935 (+ 3184us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:50:28.640940 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:50:28.647107 (+ 6167us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:50:28.647129 (+ 22us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:50:28.649470 (+ 2341us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:50:28.649477 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:50:28.649691 (+ 214us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:50:28.649756 (+ 65us) client_negotiation.cc:769] Sending connection context
0620 14:50:28.649887 (+ 131us) client_negotiation.cc:240] Negotiation successful
0620 14:50:28.650151 (+ 264us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":13602,"thread_start_us":13397,"threads_started":1}
I20260620 14:50:28.652058 13792 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:50:28.623111 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.3:44975 (local address 127.0.0.1:38594)
0620 14:50:28.636042 (+ 12931us) negotiation.cc:107] Waiting for socket to connect
0620 14:50:28.636066 (+ 24us) client_negotiation.cc:174] Beginning negotiation
0620 14:50:28.636218 (+ 152us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 14:50:28.639511 (+ 3293us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 14:50:28.639516 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 14:50:28.639540 (+ 24us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 14:50:28.640000 (+ 460us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:50:28.640012 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:50:28.642172 (+ 2160us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:50:28.642179 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:50:28.647790 (+ 5611us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:50:28.647806 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:50:28.650060 (+ 2254us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:50:28.650067 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:50:28.650241 (+ 174us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:50:28.650276 (+ 35us) client_negotiation.cc:769] Sending connection context
0620 14:50:28.650356 (+ 80us) client_negotiation.cc:240] Negotiation successful
0620 14:50:28.650488 (+ 132us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":12742,"thread_start_us":17186,"threads_started":1}
I20260620 14:50:28.656400 13427 tablet_service.cc:1467] Processing CreateTablet for tablet befa054776a244069eb408d51da07fd3 (DEFAULT_TABLE table=test-table [id=af42d07a998e4804a180a09710eb10fe]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 14:50:28.656381 13706 tablet_service.cc:1467] Processing CreateTablet for tablet befa054776a244069eb408d51da07fd3 (DEFAULT_TABLE table=test-table [id=af42d07a998e4804a180a09710eb10fe]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 14:50:28.657909 13706 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet befa054776a244069eb408d51da07fd3. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 14:50:28.657920 13427 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet befa054776a244069eb408d51da07fd3. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 14:50:28.671939 13789 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:50:28.623465 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.2:34423 (local address 127.0.0.1:40452)
0620 14:50:28.640603 (+ 17138us) negotiation.cc:107] Waiting for socket to connect
0620 14:50:28.640633 (+ 30us) client_negotiation.cc:174] Beginning negotiation
0620 14:50:28.640757 (+ 124us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 14:50:28.653775 (+ 13018us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 14:50:28.653781 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 14:50:28.653813 (+ 32us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 14:50:28.654242 (+ 429us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:50:28.654255 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:50:28.666368 (+ 12113us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:50:28.666375 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:50:28.668593 (+ 2218us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:50:28.668620 (+ 27us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:50:28.671026 (+ 2406us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:50:28.671032 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:50:28.671215 (+ 183us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:50:28.671252 (+ 37us) client_negotiation.cc:769] Sending connection context
0620 14:50:28.671365 (+ 113us) client_negotiation.cc:240] Negotiation successful
0620 14:50:28.671535 (+ 170us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":17002,"thread_start_us":12213,"threads_started":1}
I20260620 14:50:28.671996 13791 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:50:28.624046 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40452 (local address 127.0.97.2:34423)
0620 14:50:28.636922 (+ 12876us) server_negotiation.cc:206] Beginning negotiation
0620 14:50:28.636935 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0620 14:50:28.640766 (+ 3831us) server_negotiation.cc:407] Connection header received
0620 14:50:28.640892 (+ 126us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:50:28.640901 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:50:28.641194 (+ 293us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:50:28.653283 (+ 12089us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 14:50:28.664337 (+ 11054us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:50:28.666087 (+ 1750us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:50:28.668850 (+ 2763us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:50:28.670423 (+ 1573us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:50:28.670575 (+ 152us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:50:28.670824 (+ 249us) server_negotiation.cc:1035] Waiting for connection context
0620 14:50:28.671430 (+ 606us) server_negotiation.cc:299] Negotiation successful
0620 14:50:28.671653 (+ 223us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":12636,"thread_start_us":12413,"threads_started":1}
I20260620 14:50:28.673831 13799 tablet_bootstrap.cc:492] T befa054776a244069eb408d51da07fd3 P 98035054f3114f6e900361e476f372db: Bootstrap starting.
I20260620 14:50:28.676507 13800 tablet_bootstrap.cc:492] T befa054776a244069eb408d51da07fd3 P a71cb9da2bb048879d6dac6997bcf785: Bootstrap starting.
I20260620 14:50:28.676599 13571 tablet_service.cc:1467] Processing CreateTablet for tablet befa054776a244069eb408d51da07fd3 (DEFAULT_TABLE table=test-table [id=af42d07a998e4804a180a09710eb10fe]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 14:50:28.678045 13799 tablet_bootstrap.cc:654] T befa054776a244069eb408d51da07fd3 P 98035054f3114f6e900361e476f372db: Neither blocks nor log segments found. Creating new log.
I20260620 14:50:28.679137 13571 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet befa054776a244069eb408d51da07fd3. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 14:50:28.679421 13799 log.cc:824] T befa054776a244069eb408d51da07fd3 P 98035054f3114f6e900361e476f372db: Log is configured to *not* fsync() on all Append() calls
I20260620 14:50:28.680380 13800 tablet_bootstrap.cc:654] T befa054776a244069eb408d51da07fd3 P a71cb9da2bb048879d6dac6997bcf785: Neither blocks nor log segments found. Creating new log.
I20260620 14:50:28.681650 13800 log.cc:824] T befa054776a244069eb408d51da07fd3 P a71cb9da2bb048879d6dac6997bcf785: Log is configured to *not* fsync() on all Append() calls
I20260620 14:50:28.682895 13799 tablet_bootstrap.cc:492] T befa054776a244069eb408d51da07fd3 P 98035054f3114f6e900361e476f372db: No bootstrap required, opened a new log
I20260620 14:50:28.683216 13799 ts_tablet_manager.cc:1397] T befa054776a244069eb408d51da07fd3 P 98035054f3114f6e900361e476f372db: Time spent bootstrapping tablet: real 0.010s user 0.003s sys 0.004s
I20260620 14:50:28.684970 13800 tablet_bootstrap.cc:492] T befa054776a244069eb408d51da07fd3 P a71cb9da2bb048879d6dac6997bcf785: No bootstrap required, opened a new log
I20260620 14:50:28.685264 13800 ts_tablet_manager.cc:1397] T befa054776a244069eb408d51da07fd3 P a71cb9da2bb048879d6dac6997bcf785: Time spent bootstrapping tablet: real 0.009s user 0.007s sys 0.000s
I20260620 14:50:28.692605 13803 tablet_bootstrap.cc:492] T befa054776a244069eb408d51da07fd3 P 4868180fab3242e691c013a76dd5cb47: Bootstrap starting.
I20260620 14:50:28.694156 13799 raft_consensus.cc:348] T befa054776a244069eb408d51da07fd3 P 98035054f3114f6e900361e476f372db [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a71cb9da2bb048879d6dac6997bcf785" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 42323 } } peers { permanent_uuid: "98035054f3114f6e900361e476f372db" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 44975 } } peers { permanent_uuid: "4868180fab3242e691c013a76dd5cb47" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 34423 } }
I20260620 14:50:28.694645 13799 raft_consensus.cc:374] T befa054776a244069eb408d51da07fd3 P 98035054f3114f6e900361e476f372db [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 14:50:28.694742 13799 raft_consensus.cc:729] T befa054776a244069eb408d51da07fd3 P 98035054f3114f6e900361e476f372db [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 98035054f3114f6e900361e476f372db, State: Initialized, Role: FOLLOWER
I20260620 14:50:28.695387 13799 consensus_queue.cc:260] T befa054776a244069eb408d51da07fd3 P 98035054f3114f6e900361e476f372db [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a71cb9da2bb048879d6dac6997bcf785" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 42323 } } peers { permanent_uuid: "98035054f3114f6e900361e476f372db" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 44975 } } peers { permanent_uuid: "4868180fab3242e691c013a76dd5cb47" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 34423 } }
I20260620 14:50:28.696074 13803 tablet_bootstrap.cc:654] T befa054776a244069eb408d51da07fd3 P 4868180fab3242e691c013a76dd5cb47: Neither blocks nor log segments found. Creating new log.
I20260620 14:50:28.696053 13800 raft_consensus.cc:348] T befa054776a244069eb408d51da07fd3 P a71cb9da2bb048879d6dac6997bcf785 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a71cb9da2bb048879d6dac6997bcf785" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 42323 } } peers { permanent_uuid: "98035054f3114f6e900361e476f372db" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 44975 } } peers { permanent_uuid: "4868180fab3242e691c013a76dd5cb47" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 34423 } }
I20260620 14:50:28.696621 13800 raft_consensus.cc:374] T befa054776a244069eb408d51da07fd3 P a71cb9da2bb048879d6dac6997bcf785 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 14:50:28.696753 13800 raft_consensus.cc:729] T befa054776a244069eb408d51da07fd3 P a71cb9da2bb048879d6dac6997bcf785 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a71cb9da2bb048879d6dac6997bcf785, State: Initialized, Role: FOLLOWER
I20260620 14:50:28.697346 13803 log.cc:824] T befa054776a244069eb408d51da07fd3 P 4868180fab3242e691c013a76dd5cb47: Log is configured to *not* fsync() on all Append() calls
I20260620 14:50:28.697618 13800 consensus_queue.cc:260] T befa054776a244069eb408d51da07fd3 P a71cb9da2bb048879d6dac6997bcf785 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a71cb9da2bb048879d6dac6997bcf785" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 42323 } } peers { permanent_uuid: "98035054f3114f6e900361e476f372db" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 44975 } } peers { permanent_uuid: "4868180fab3242e691c013a76dd5cb47" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 34423 } }
I20260620 14:50:28.700555 13492 heartbeater.cc:503] Master 127.0.97.62:38829 was elected leader, sending a full tablet report...
I20260620 14:50:28.701853 13800 ts_tablet_manager.cc:1428] T befa054776a244069eb408d51da07fd3 P a71cb9da2bb048879d6dac6997bcf785: Time spent starting tablet: real 0.016s user 0.014s sys 0.002s
I20260620 14:50:28.702903 13771 heartbeater.cc:503] Master 127.0.97.62:38829 was elected leader, sending a full tablet report...
I20260620 14:50:28.703616 13799 ts_tablet_manager.cc:1428] T befa054776a244069eb408d51da07fd3 P 98035054f3114f6e900361e476f372db: Time spent starting tablet: real 0.020s user 0.013s sys 0.009s
I20260620 14:50:28.708009 13803 tablet_bootstrap.cc:492] T befa054776a244069eb408d51da07fd3 P 4868180fab3242e691c013a76dd5cb47: No bootstrap required, opened a new log
I20260620 14:50:28.708349 13803 ts_tablet_manager.cc:1397] T befa054776a244069eb408d51da07fd3 P 4868180fab3242e691c013a76dd5cb47: Time spent bootstrapping tablet: real 0.016s user 0.006s sys 0.008s
I20260620 14:50:28.715847 13803 raft_consensus.cc:348] T befa054776a244069eb408d51da07fd3 P 4868180fab3242e691c013a76dd5cb47 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a71cb9da2bb048879d6dac6997bcf785" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 42323 } } peers { permanent_uuid: "98035054f3114f6e900361e476f372db" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 44975 } } peers { permanent_uuid: "4868180fab3242e691c013a76dd5cb47" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 34423 } }
I20260620 14:50:28.716264 13803 raft_consensus.cc:374] T befa054776a244069eb408d51da07fd3 P 4868180fab3242e691c013a76dd5cb47 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 14:50:28.716349 13803 raft_consensus.cc:729] T befa054776a244069eb408d51da07fd3 P 4868180fab3242e691c013a76dd5cb47 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4868180fab3242e691c013a76dd5cb47, State: Initialized, Role: FOLLOWER
I20260620 14:50:28.716879 13803 consensus_queue.cc:260] T befa054776a244069eb408d51da07fd3 P 4868180fab3242e691c013a76dd5cb47 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a71cb9da2bb048879d6dac6997bcf785" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 42323 } } peers { permanent_uuid: "98035054f3114f6e900361e476f372db" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 44975 } } peers { permanent_uuid: "4868180fab3242e691c013a76dd5cb47" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 34423 } }
I20260620 14:50:28.718380 13803 ts_tablet_manager.cc:1428] T befa054776a244069eb408d51da07fd3 P 4868180fab3242e691c013a76dd5cb47: Time spent starting tablet: real 0.010s user 0.010s sys 0.000s
W20260620 14:50:28.745115 13772 tablet.cc:2369] T befa054776a244069eb408d51da07fd3 P 98035054f3114f6e900361e476f372db: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260620 14:50:28.775576 13637 tablet.cc:2369] T befa054776a244069eb408d51da07fd3 P 4868180fab3242e691c013a76dd5cb47: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260620 14:50:28.888949 13493 tablet.cc:2369] T befa054776a244069eb408d51da07fd3 P a71cb9da2bb048879d6dac6997bcf785: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260620 14:50:28.905251 13807 raft_consensus.cc:482] T befa054776a244069eb408d51da07fd3 P 4868180fab3242e691c013a76dd5cb47 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260620 14:50:28.905577 13807 raft_consensus.cc:504] T befa054776a244069eb408d51da07fd3 P 4868180fab3242e691c013a76dd5cb47 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a71cb9da2bb048879d6dac6997bcf785" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 42323 } } peers { permanent_uuid: "98035054f3114f6e900361e476f372db" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 44975 } } peers { permanent_uuid: "4868180fab3242e691c013a76dd5cb47" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 34423 } }
I20260620 14:50:28.907354 13807 leader_election.cc:288] T befa054776a244069eb408d51da07fd3 P 4868180fab3242e691c013a76dd5cb47 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers a71cb9da2bb048879d6dac6997bcf785 (127.0.97.1:42323), 98035054f3114f6e900361e476f372db (127.0.97.3:44975)
I20260620 14:50:28.914525 13790 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:50:28.907963 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.2:60647 (local address 127.0.97.3:44975)
0620 14:50:28.908194 (+ 231us) server_negotiation.cc:206] Beginning negotiation
0620 14:50:28.908203 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0620 14:50:28.908975 (+ 772us) server_negotiation.cc:407] Connection header received
0620 14:50:28.909175 (+ 200us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:50:28.909192 (+ 17us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:50:28.909252 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:50:28.909347 (+ 95us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 14:50:28.910008 (+ 661us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:50:28.910744 (+ 736us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:50:28.912664 (+ 1920us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:50:28.913440 (+ 776us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:50:28.913607 (+ 167us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:50:28.913726 (+ 119us) server_negotiation.cc:1035] Waiting for connection context
0620 14:50:28.914088 (+ 362us) server_negotiation.cc:299] Negotiation successful
0620 14:50:28.914248 (+ 160us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":75}
I20260620 14:50:28.914541 13787 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:50:28.907882 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.2:34895 (local address 127.0.97.1:42323)
0620 14:50:28.908156 (+ 274us) server_negotiation.cc:206] Beginning negotiation
0620 14:50:28.908166 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0620 14:50:28.908879 (+ 713us) server_negotiation.cc:407] Connection header received
0620 14:50:28.909118 (+ 239us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:50:28.909122 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:50:28.909193 (+ 71us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:50:28.909294 (+ 101us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 14:50:28.909941 (+ 647us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:50:28.910673 (+ 732us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:50:28.912655 (+ 1982us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:50:28.913428 (+ 773us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:50:28.913606 (+ 178us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:50:28.913738 (+ 132us) server_negotiation.cc:1035] Waiting for connection context
0620 14:50:28.914096 (+ 358us) server_negotiation.cc:299] Negotiation successful
0620 14:50:28.914243 (+ 147us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":85}
I20260620 14:50:28.914713 13808 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:50:28.907878 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.3:44975 (local address 127.0.97.2:60647)
0620 14:50:28.908860 (+ 982us) negotiation.cc:107] Waiting for socket to connect
0620 14:50:28.908875 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0620 14:50:28.909033 (+ 158us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 14:50:28.909362 (+ 329us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 14:50:28.909367 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 14:50:28.909389 (+ 22us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 14:50:28.909839 (+ 450us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:50:28.909849 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:50:28.910888 (+ 1039us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:50:28.910900 (+ 12us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:50:28.912503 (+ 1603us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:50:28.912516 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:50:28.913707 (+ 1191us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:50:28.913718 (+ 11us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:50:28.913910 (+ 192us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:50:28.913943 (+ 33us) client_negotiation.cc:769] Sending connection context
0620 14:50:28.914060 (+ 117us) client_negotiation.cc:240] Negotiation successful
0620 14:50:28.914248 (+ 188us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":873,"mutex_wait_us":86,"thread_start_us":513,"threads_started":1}
I20260620 14:50:28.915082 13809 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:50:28.907827 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:42323 (local address 127.0.97.2:34895)
0620 14:50:28.908760 (+ 933us) negotiation.cc:107] Waiting for socket to connect
0620 14:50:28.908776 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0620 14:50:28.908959 (+ 183us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 14:50:28.909309 (+ 350us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 14:50:28.909313 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 14:50:28.909337 (+ 24us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 14:50:28.909791 (+ 454us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:50:28.909802 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:50:28.910804 (+ 1002us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:50:28.910812 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:50:28.912493 (+ 1681us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:50:28.912506 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:50:28.913739 (+ 1233us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:50:28.913746 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:50:28.913914 (+ 168us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:50:28.913944 (+ 30us) client_negotiation.cc:769] Sending connection context
0620 14:50:28.914066 (+ 122us) client_negotiation.cc:240] Negotiation successful
0620 14:50:28.914243 (+ 177us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":786,"mutex_wait_us":107,"thread_start_us":594,"threads_started":1}
I20260620 14:50:28.915771 13726 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "befa054776a244069eb408d51da07fd3" candidate_uuid: "4868180fab3242e691c013a76dd5cb47" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "98035054f3114f6e900361e476f372db" is_pre_election: true
I20260620 14:50:28.916124 13447 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "befa054776a244069eb408d51da07fd3" candidate_uuid: "4868180fab3242e691c013a76dd5cb47" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a71cb9da2bb048879d6dac6997bcf785" is_pre_election: true
I20260620 14:50:28.916733 13447 raft_consensus.cc:2454] T befa054776a244069eb408d51da07fd3 P a71cb9da2bb048879d6dac6997bcf785 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4868180fab3242e691c013a76dd5cb47 in term 0.
I20260620 14:50:28.917301 13726 raft_consensus.cc:2454] T befa054776a244069eb408d51da07fd3 P 98035054f3114f6e900361e476f372db [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4868180fab3242e691c013a76dd5cb47 in term 0.
I20260620 14:50:28.917626 13524 leader_election.cc:302] T befa054776a244069eb408d51da07fd3 P 4868180fab3242e691c013a76dd5cb47 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4868180fab3242e691c013a76dd5cb47, a71cb9da2bb048879d6dac6997bcf785; no voters:
I20260620 14:50:28.918072 13807 raft_consensus.cc:2781] T befa054776a244069eb408d51da07fd3 P 4868180fab3242e691c013a76dd5cb47 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260620 14:50:28.918243 13807 raft_consensus.cc:482] T befa054776a244069eb408d51da07fd3 P 4868180fab3242e691c013a76dd5cb47 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260620 14:50:28.918352 13807 raft_consensus.cc:3037] T befa054776a244069eb408d51da07fd3 P 4868180fab3242e691c013a76dd5cb47 [term 0 FOLLOWER]: Advancing to term 1
I20260620 14:50:28.920590 13807 raft_consensus.cc:504] T befa054776a244069eb408d51da07fd3 P 4868180fab3242e691c013a76dd5cb47 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a71cb9da2bb048879d6dac6997bcf785" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 42323 } } peers { permanent_uuid: "98035054f3114f6e900361e476f372db" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 44975 } } peers { permanent_uuid: "4868180fab3242e691c013a76dd5cb47" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 34423 } }
I20260620 14:50:28.921169 13807 leader_election.cc:288] T befa054776a244069eb408d51da07fd3 P 4868180fab3242e691c013a76dd5cb47 [CANDIDATE]: Term 1 election: Requested vote from peers a71cb9da2bb048879d6dac6997bcf785 (127.0.97.1:42323), 98035054f3114f6e900361e476f372db (127.0.97.3:44975)
I20260620 14:50:28.921654 13726 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "befa054776a244069eb408d51da07fd3" candidate_uuid: "4868180fab3242e691c013a76dd5cb47" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "98035054f3114f6e900361e476f372db"
I20260620 14:50:28.921725 13447 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "befa054776a244069eb408d51da07fd3" candidate_uuid: "4868180fab3242e691c013a76dd5cb47" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a71cb9da2bb048879d6dac6997bcf785"
I20260620 14:50:28.921852 13726 raft_consensus.cc:3037] T befa054776a244069eb408d51da07fd3 P 98035054f3114f6e900361e476f372db [term 0 FOLLOWER]: Advancing to term 1
I20260620 14:50:28.921949 13447 raft_consensus.cc:3037] T befa054776a244069eb408d51da07fd3 P a71cb9da2bb048879d6dac6997bcf785 [term 0 FOLLOWER]: Advancing to term 1
I20260620 14:50:28.924206 13726 raft_consensus.cc:2454] T befa054776a244069eb408d51da07fd3 P 98035054f3114f6e900361e476f372db [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4868180fab3242e691c013a76dd5cb47 in term 1.
I20260620 14:50:28.924427 13447 raft_consensus.cc:2454] T befa054776a244069eb408d51da07fd3 P a71cb9da2bb048879d6dac6997bcf785 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4868180fab3242e691c013a76dd5cb47 in term 1.
I20260620 14:50:28.924671 13525 leader_election.cc:302] T befa054776a244069eb408d51da07fd3 P 4868180fab3242e691c013a76dd5cb47 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4868180fab3242e691c013a76dd5cb47, 98035054f3114f6e900361e476f372db; no voters:
I20260620 14:50:28.924989 13807 raft_consensus.cc:2781] T befa054776a244069eb408d51da07fd3 P 4868180fab3242e691c013a76dd5cb47 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 14:50:28.925447 13807 raft_consensus.cc:686] T befa054776a244069eb408d51da07fd3 P 4868180fab3242e691c013a76dd5cb47 [term 1 LEADER]: Becoming Leader. State: Replica: 4868180fab3242e691c013a76dd5cb47, State: Running, Role: LEADER
I20260620 14:50:28.925943 13807 consensus_queue.cc:237] T befa054776a244069eb408d51da07fd3 P 4868180fab3242e691c013a76dd5cb47 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a71cb9da2bb048879d6dac6997bcf785" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 42323 } } peers { permanent_uuid: "98035054f3114f6e900361e476f372db" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 44975 } } peers { permanent_uuid: "4868180fab3242e691c013a76dd5cb47" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 34423 } }
I20260620 14:50:28.930517 13295 catalog_manager.cc:5495] T befa054776a244069eb408d51da07fd3 P 4868180fab3242e691c013a76dd5cb47 reported cstate change: term changed from 0 to 1, leader changed from <none> to 4868180fab3242e691c013a76dd5cb47 (127.0.97.2). New cstate: current_term: 1 leader_uuid: "4868180fab3242e691c013a76dd5cb47" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a71cb9da2bb048879d6dac6997bcf785" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 42323 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "98035054f3114f6e900361e476f372db" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 44975 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "4868180fab3242e691c013a76dd5cb47" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 34423 } health_report { overall_health: HEALTHY } } }
I20260620 14:50:28.955370 13791 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:50:28.949459 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40464 (local address 127.0.97.2:34423)
0620 14:50:28.949685 (+ 226us) server_negotiation.cc:206] Beginning negotiation
0620 14:50:28.949691 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 14:50:28.949713 (+ 22us) server_negotiation.cc:407] Connection header received
0620 14:50:28.949855 (+ 142us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:50:28.949859 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:50:28.949953 (+ 94us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:50:28.950110 (+ 157us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 14:50:28.950487 (+ 377us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:50:28.951229 (+ 742us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:50:28.952756 (+ 1527us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:50:28.953170 (+ 414us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:50:28.953317 (+ 147us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:50:28.953872 (+ 555us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 14:50:28.954646 (+ 774us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0620 14:50:28.954806 (+ 160us) server_negotiation.cc:1035] Waiting for connection context
0620 14:50:28.954934 (+ 128us) server_negotiation.cc:299] Negotiation successful
0620 14:50:28.955044 (+ 110us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":68}
I20260620 14:50:28.965850 13726 raft_consensus.cc:1261] T befa054776a244069eb408d51da07fd3 P 98035054f3114f6e900361e476f372db [term 1 FOLLOWER]: Refusing update from remote peer 4868180fab3242e691c013a76dd5cb47: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 14:50:28.965906 13447 raft_consensus.cc:1261] T befa054776a244069eb408d51da07fd3 P a71cb9da2bb048879d6dac6997bcf785 [term 1 FOLLOWER]: Refusing update from remote peer 4868180fab3242e691c013a76dd5cb47: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 14:50:28.966933 13810 consensus_queue.cc:1035] T befa054776a244069eb408d51da07fd3 P 4868180fab3242e691c013a76dd5cb47 [LEADER]: Connected to new peer: Peer: permanent_uuid: "98035054f3114f6e900361e476f372db" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 44975 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 14:50:28.967383 13807 consensus_queue.cc:1035] T befa054776a244069eb408d51da07fd3 P 4868180fab3242e691c013a76dd5cb47 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a71cb9da2bb048879d6dac6997bcf785" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 42323 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 14:50:28.979223 13813 mvcc.cc:205] Tried to move back new op lower bound from 7298936950632099840 to 7298936950484656128. Current Snapshot: MvccSnapshot[applied={T|T < 7298936950632099840}]
I20260620 14:50:28.988807 13815 mvcc.cc:205] Tried to move back new op lower bound from 7298936950632099840 to 7298936950484656128. Current Snapshot: MvccSnapshot[applied={T|T < 7298936950632099840}]
I20260620 14:50:28.994993 13295 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:40698:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 14:50:28.995288 13295 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:40698:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 14:50:29.000020 13295 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 384f5045b6a1446b8de95d923f9e10c2: Sending DeleteTablet for 3 replicas of tablet befa054776a244069eb408d51da07fd3
I20260620 14:50:29.005342 13706 tablet_service.cc:1514] Processing DeleteTablet for tablet befa054776a244069eb408d51da07fd3 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 14:50:28 UTC) from {username='kudu', principal='kudu/127.0.97.62@KRBTEST.COM'} at 127.0.0.1:38594
I20260620 14:50:29.006350 13822 tablet_replica.cc:330] T befa054776a244069eb408d51da07fd3 P 98035054f3114f6e900361e476f372db: stopping tablet replica
I20260620 14:50:29.006870 13822 raft_consensus.cc:2229] T befa054776a244069eb408d51da07fd3 P 98035054f3114f6e900361e476f372db [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 14:50:29.007426 13822 raft_consensus.cc:2258] T befa054776a244069eb408d51da07fd3 P 98035054f3114f6e900361e476f372db [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 14:50:29.007943 13427 tablet_service.cc:1514] Processing DeleteTablet for tablet befa054776a244069eb408d51da07fd3 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 14:50:28 UTC) from {username='kudu', principal='kudu/127.0.97.62@KRBTEST.COM'} at 127.0.0.1:41100
I20260620 14:50:29.009662 13822 ts_tablet_manager.cc:1905] T befa054776a244069eb408d51da07fd3 P 98035054f3114f6e900361e476f372db: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 14:50:29.010524 13571 tablet_service.cc:1514] Processing DeleteTablet for tablet befa054776a244069eb408d51da07fd3 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 14:50:28 UTC) from {username='kudu', principal='kudu/127.0.97.62@KRBTEST.COM'} at 127.0.0.1:40452
I20260620 14:50:29.019115 13822 ts_tablet_manager.cc:1918] T befa054776a244069eb408d51da07fd3 P 98035054f3114f6e900361e476f372db: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260620 14:50:29.019837 13823 tablet_replica.cc:330] T befa054776a244069eb408d51da07fd3 P a71cb9da2bb048879d6dac6997bcf785: stopping tablet replica
I20260620 14:50:29.020387 13823 raft_consensus.cc:2229] T befa054776a244069eb408d51da07fd3 P a71cb9da2bb048879d6dac6997bcf785 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 14:50:29.020967 13823 raft_consensus.cc:2258] T befa054776a244069eb408d51da07fd3 P a71cb9da2bb048879d6dac6997bcf785 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 14:50:29.021045 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskUci3DU/build/asan/bin/kudu pid 13332
I20260620 14:50:29.024045 13823 ts_tablet_manager.cc:1905] T befa054776a244069eb408d51da07fd3 P a71cb9da2bb048879d6dac6997bcf785: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 14:50:29.026566 13824 tablet_replica.cc:330] T befa054776a244069eb408d51da07fd3 P 4868180fab3242e691c013a76dd5cb47: stopping tablet replica
I20260620 14:50:29.028323 13824 raft_consensus.cc:2229] T befa054776a244069eb408d51da07fd3 P 4868180fab3242e691c013a76dd5cb47 [term 1 LEADER]: Raft consensus shutting down.
I20260620 14:50:29.029043 13824 raft_consensus.cc:2258] T befa054776a244069eb408d51da07fd3 P 4868180fab3242e691c013a76dd5cb47 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 14:50:29.019798 13706 tablet_service.cc:1514] Processing DeleteTablet for tablet befa054776a244069eb408d51da07fd3 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 14:50:28 UTC) from {username='kudu', principal='kudu/127.0.97.62@KRBTEST.COM'} at 127.0.0.1:38594
I20260620 14:50:29.031653 13822 log.cc:1196] T befa054776a244069eb408d51da07fd3 P 98035054f3114f6e900361e476f372db: Deleting WAL directory at /tmp/dist-test-taskUci3DU/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781966829091300-388-0/minicluster-data/ts-2/wal/wals/befa054776a244069eb408d51da07fd3
I20260620 14:50:29.032280 13822 ts_tablet_manager.cc:1939] T befa054776a244069eb408d51da07fd3 P 98035054f3114f6e900361e476f372db: Deleting consensus metadata
I20260620 14:50:29.035008 13823 ts_tablet_manager.cc:1918] T befa054776a244069eb408d51da07fd3 P a71cb9da2bb048879d6dac6997bcf785: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260620 14:50:29.035241 13823 log.cc:1196] T befa054776a244069eb408d51da07fd3 P a71cb9da2bb048879d6dac6997bcf785: Deleting WAL directory at /tmp/dist-test-taskUci3DU/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781966829091300-388-0/minicluster-data/ts-0/wal/wals/befa054776a244069eb408d51da07fd3
I20260620 14:50:29.035872 13823 ts_tablet_manager.cc:1939] T befa054776a244069eb408d51da07fd3 P a71cb9da2bb048879d6dac6997bcf785: Deleting consensus metadata
I20260620 14:50:29.036868 13280 catalog_manager.cc:4841] TS 98035054f3114f6e900361e476f372db (127.0.97.3:44975): tablet befa054776a244069eb408d51da07fd3 (table test-table [id=af42d07a998e4804a180a09710eb10fe]) successfully deleted
W20260620 14:50:29.037438 13280 catalog_manager.cc:4808] TS 98035054f3114f6e900361e476f372db (127.0.97.3:44975): delete failed for tablet befa054776a244069eb408d51da07fd3 because the tablet was not found. No further retry: Not found: Tablet not found: befa054776a244069eb408d51da07fd3
I20260620 14:50:29.037737 13279 catalog_manager.cc:4841] TS a71cb9da2bb048879d6dac6997bcf785 (127.0.97.1:42323): tablet befa054776a244069eb408d51da07fd3 (table test-table [id=af42d07a998e4804a180a09710eb10fe]) successfully deleted
I20260620 14:50:29.041380 13824 ts_tablet_manager.cc:1905] T befa054776a244069eb408d51da07fd3 P 4868180fab3242e691c013a76dd5cb47: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 14:50:29.050549 13824 ts_tablet_manager.cc:1918] T befa054776a244069eb408d51da07fd3 P 4868180fab3242e691c013a76dd5cb47: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260620 14:50:29.050897 13824 log.cc:1196] T befa054776a244069eb408d51da07fd3 P 4868180fab3242e691c013a76dd5cb47: Deleting WAL directory at /tmp/dist-test-taskUci3DU/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781966829091300-388-0/minicluster-data/ts-1/wal/wals/befa054776a244069eb408d51da07fd3
I20260620 14:50:29.051020 13787 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:50:29.023240 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41112 (local address 127.0.97.1:42323)
0620 14:50:29.023431 (+ 191us) server_negotiation.cc:206] Beginning negotiation
0620 14:50:29.023438 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0620 14:50:29.023459 (+ 21us) server_negotiation.cc:407] Connection header received
0620 14:50:29.023526 (+ 67us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:50:29.023533 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:50:29.023647 (+ 114us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:50:29.023770 (+ 123us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:50:29.025920 (+ 2150us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:50:29.027167 (+ 1247us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:50:29.031437 (+ 4270us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:50:29.032293 (+ 856us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:50:29.032442 (+ 149us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:50:29.040632 (+ 8190us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 14:50:29.040651 (+ 19us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 14:50:29.040672 (+ 21us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 14:50:29.040739 (+ 67us) server_negotiation.cc:922] calling sasl_server_start()
0620 14:50:29.045988 (+ 5249us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:50:29.047186 (+ 1198us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:50:29.047199 (+ 13us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:50:29.047205 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:50:29.047398 (+ 193us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:50:29.047937 (+ 539us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:50:29.047944 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:50:29.047950 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:50:29.048475 (+ 525us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 14:50:29.048819 (+ 344us) server_negotiation.cc:1035] Waiting for connection context
0620 14:50:29.050247 (+ 1428us) server_negotiation.cc:299] Negotiation successful
0620 14:50:29.050517 (+ 270us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":66}
I20260620 14:50:29.051872 13487 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 14:50:29.051903 13824 ts_tablet_manager.cc:1939] T befa054776a244069eb408d51da07fd3 P 4868180fab3242e691c013a76dd5cb47: Deleting consensus metadata
I20260620 14:50:29.054296 13282 catalog_manager.cc:4841] TS 4868180fab3242e691c013a76dd5cb47 (127.0.97.2:34423): tablet befa054776a244069eb408d51da07fd3 (table test-table [id=af42d07a998e4804a180a09710eb10fe]) successfully deleted
I20260620 14:50:29.186868 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskUci3DU/build/asan/bin/kudu with pid 13332
I20260620 14:50:29.203970 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskUci3DU/build/asan/bin/kudu pid 13496
I20260620 14:50:29.218726 13791 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:50:29.204893 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40478 (local address 127.0.97.2:34423)
0620 14:50:29.205410 (+ 517us) server_negotiation.cc:206] Beginning negotiation
0620 14:50:29.205415 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 14:50:29.205430 (+ 15us) server_negotiation.cc:407] Connection header received
0620 14:50:29.205487 (+ 57us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:50:29.205491 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:50:29.205558 (+ 67us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:50:29.205673 (+ 115us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:50:29.206970 (+ 1297us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:50:29.207867 (+ 897us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:50:29.209265 (+ 1398us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:50:29.209737 (+ 472us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:50:29.209884 (+ 147us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:50:29.212606 (+ 2722us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 14:50:29.212624 (+ 18us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 14:50:29.212640 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 14:50:29.212696 (+ 56us) server_negotiation.cc:922] calling sasl_server_start()
0620 14:50:29.215810 (+ 3114us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:50:29.216514 (+ 704us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:50:29.216523 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:50:29.216525 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:50:29.216674 (+ 149us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:50:29.217076 (+ 402us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:50:29.217082 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:50:29.217089 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:50:29.217462 (+ 373us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 14:50:29.217826 (+ 364us) server_negotiation.cc:1035] Waiting for connection context
0620 14:50:29.218103 (+ 277us) server_negotiation.cc:299] Negotiation successful
0620 14:50:29.218349 (+ 246us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":401}
I20260620 14:50:29.219412 13631 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 14:50:29.334596 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskUci3DU/build/asan/bin/kudu with pid 13496
I20260620 14:50:29.353305 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskUci3DU/build/asan/bin/kudu pid 13640
I20260620 14:50:29.367165 13790 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:50:29.354103 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38602 (local address 127.0.97.3:44975)
0620 14:50:29.354300 (+ 197us) server_negotiation.cc:206] Beginning negotiation
0620 14:50:29.354304 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 14:50:29.354324 (+ 20us) server_negotiation.cc:407] Connection header received
0620 14:50:29.354499 (+ 175us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:50:29.354503 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:50:29.354577 (+ 74us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:50:29.354660 (+ 83us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:50:29.355971 (+ 1311us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:50:29.356725 (+ 754us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:50:29.358175 (+ 1450us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:50:29.358629 (+ 454us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:50:29.358730 (+ 101us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:50:29.361470 (+ 2740us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 14:50:29.361484 (+ 14us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 14:50:29.361500 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 14:50:29.361535 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0620 14:50:29.364537 (+ 3002us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:50:29.365234 (+ 697us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:50:29.365241 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:50:29.365243 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:50:29.365368 (+ 125us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:50:29.365696 (+ 328us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:50:29.365699 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:50:29.365701 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:50:29.366035 (+ 334us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 14:50:29.366275 (+ 240us) server_negotiation.cc:1035] Waiting for connection context
0620 14:50:29.366613 (+ 338us) server_negotiation.cc:299] Negotiation successful
0620 14:50:29.366794 (+ 181us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":69}
I20260620 14:50:29.368196 13766 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 14:50:29.459393 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskUci3DU/build/asan/bin/kudu with pid 13640
I20260620 14:50:29.476014 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskUci3DU/build/asan/bin/kudu pid 13262
I20260620 14:50:29.477308 13325 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 14:50:29.591871 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskUci3DU/build/asan/bin/kudu with pid 13262
I20260620 14:50:29.615279 388 mini_ranger_kms.cc:60] Stopping Ranger KMS...
I20260620 14:50:29.951490 388 mini_ranger_kms.cc:62] Stopped Ranger KMS
I20260620 14:50:29.951769 388 mini_ranger.cc:66] Stopping Ranger...
I20260620 14:50:30.299861 388 mini_ranger.cc:68] Stopped Ranger
2026-06-20 14:50:30.300 UTC [12314] LOG: received smart shutdown request
2026-06-20 14:50:30.302 UTC [12314] LOG: background worker "logical replication launcher" (PID 12323) exited with exit code 1
2026-06-20 14:50:30.303 UTC [12318] LOG: shutting down
2026-06-20 14:50:30.456 UTC [12314] LOG: database system is shut down
2026-06-20T14:50:30Z chronyd exiting
[ OK ] SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers (53163 ms)
[ RUN ] SecurityITest.IPKICACert
Loading random data
Initializing database '/tmp/dist-test-taskUci3DU/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781966829091300-388-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 20 14:50:30 dist-test-slave-q9j6 krb5kdc[13837](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 20 14:50:30 dist-test-slave-q9j6 krb5kdc[13837](info): set up 2 sockets
Jun 20 14:50:30 dist-test-slave-q9j6 krb5kdc[13837](info): commencing operation
krb5kdc: starting...
W20260620 14:50:31.688107 388 mini_kdc.cc:121] Time spent starting KDC: real 1.059s user 0.004s sys 0.021s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskUci3DU/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781966829091300-388-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskUci3DU/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781966829091300-388-0/krb5kdc/test-user.keytab.
Jun 20 14:50:31 dist-test-slave-q9j6 krb5kdc[13837](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781967031, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-20T14:50:31Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T14:50:31Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.97.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskUci3DU/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781966829091300-388-0/krb5kdc/kudu_127.0.97.62.keytab.
Entry for principal kudu/127.0.97.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskUci3DU/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781966829091300-388-0/krb5kdc/kudu_127.0.97.62.keytab.
I20260620 14:50:31.926827 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskUci3DU/build/asan/bin/kudu
/tmp/dist-test-taskUci3DU/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskUci3DU/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781966829091300-388-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskUci3DU/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781966829091300-388-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskUci3DU/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781966829091300-388-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskUci3DU/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781966829091300-388-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:43801
--webserver_interface=127.0.97.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskUci3DU/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781966829091300-388-0/krb5kdc/kudu_127.0.97.62.keytab
--principal=kudu/127.0.97.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:34493
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.97.62:43801,127.0.97.61:36315,127.0.97.60:45215
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskUci3DU/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781966829091300-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskUci3DU/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781966829091300-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskUci3DU/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781966829091300-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 14:50:32.271950 13850 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 14:50:32.272274 13850 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 14:50:32.272496 13850 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 14:50:32.280073 13850 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 14:50:32.280200 13850 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 14:50:32.280311 13850 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 14:50:32.280418 13850 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 14:50:32.280493 13850 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 14:50:32.290750 13850 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:34493
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskUci3DU/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781966829091300-388-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskUci3DU/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781966829091300-388-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.97.62:43801,127.0.97.61:36315,127.0.97.60:45215
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskUci3DU/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781966829091300-388-0/krb5kdc/kudu_127.0.97.62.keytab
--principal=kudu/127.0.97.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:43801
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskUci3DU/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781966829091300-388-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.97.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskUci3DU/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781966829091300-388-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 13:41:20 UTC on ccycloud.quasar-vdagoi.root.comops.site
build id 4457462
ASAN enabled
I20260620 14:50:32.292261 13850 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 14:50:32.293694 13850 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 14:50:32.306128 13856 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 14:50:32.306248 13855 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 14:50:32.306265 13858 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 14:50:32.307444 13850 server_base.cc:1034] running on GCE node
I20260620 14:50:32.308385 13850 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 14:50:32.309975 13850 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 14:50:32.311199 13850 hybrid_clock.cc:648] HybridClock initialized: now 1781967032311150 us; error 38 us; skew 500 ppm
I20260620 14:50:32.311645 13850 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 14:50:32 dist-test-slave-q9j6 krb5kdc[13837](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781967032, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 14:50:32.319057 13850 init.cc:377] Logged in from keytab as kudu/127.0.97.62@KRBTEST.COM (short username kudu)
I20260620 14:50:32.320789 13850 webserver.cc:460] Webserver started at http://127.0.97.62:35765/ using document root <none> and password file <none>
I20260620 14:50:32.321592 13850 fs_manager.cc:359] Metadata directory not provided
I20260620 14:50:32.321753 13850 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 14:50:32.322085 13850 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 14:50:32.325428 13850 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskUci3DU/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781966829091300-388-0/minicluster-data/master-0/data/instance:
uuid: "a7d13d41ab9648f5b3b5884352620c8b"
format_stamp: "Formatted at 2026-06-20 14:50:32 on dist-test-slave-q9j6"
server_key: "48cb4601ce519757f31b35705c2058dc"
server_key_iv: "53aa05df1b0d50489823ee1845c01c89"
server_key_version: "encryptionkey@0"
I20260620 14:50:32.326330 13850 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskUci3DU/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781966829091300-388-0/minicluster-data/master-0/wal/instance:
uuid: "a7d13d41ab9648f5b3b5884352620c8b"
format_stamp: "Formatted at 2026-06-20 14:50:32 on dist-test-slave-q9j6"
server_key: "48cb4601ce519757f31b35705c2058dc"
server_key_iv: "53aa05df1b0d50489823ee1845c01c89"
server_key_version: "encryptionkey@0"
I20260620 14:50:32.332362 13850 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.006s sys 0.001s
I20260620 14:50:32.336263 13865 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 14:50:32.337880 13850 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.001s
I20260620 14:50:32.338096 13850 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskUci3DU/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781966829091300-388-0/minicluster-data/master-0/data,/tmp/dist-test-taskUci3DU/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781966829091300-388-0/minicluster-data/master-0/wal
uuid: "a7d13d41ab9648f5b3b5884352620c8b"
format_stamp: "Formatted at 2026-06-20 14:50:32 on dist-test-slave-q9j6"
server_key: "48cb4601ce519757f31b35705c2058dc"
server_key_iv: "53aa05df1b0d50489823ee1845c01c89"
server_key_version: "encryptionkey@0"
I20260620 14:50:32.338338 13850 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskUci3DU/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781966829091300-388-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskUci3DU/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781966829091300-388-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskUci3DU/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781966829091300-388-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 14:50:32.368644 13850 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 14:50:32.373395 13850 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 14:50:32.373692 13850 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 14:50:32.392809 13850 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.62:43801
I20260620 14:50:32.392839 13916 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.62:43801 every 8 connection(s)
I20260620 14:50:32.394330 13850 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskUci3DU/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781966829091300-388-0/minicluster-data/master-0/data/info.pb
I20260620 14:50:32.398404 13917 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 14:50:32.400859 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskUci3DU/build/asan/bin/kudu as pid 13850
I20260620 14:50:32.401081 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskUci3DU/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781966829091300-388-0/minicluster-data/master-0/wal/instance
I20260620 14:50:32.401750 388 external_mini_cluster.cc:1428] Setting key 62e16c2be47bbd7dd9311f5a760a72f6
I20260620 14:50:32.405789 13917 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.97.62" port: 43801 } has no permanent_uuid. Determining permanent_uuid...
Jun 20 14:50:32 dist-test-slave-q9j6 krb5kdc[13837](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781967032, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.62@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
WARNING: no policy specified for kudu/127.0.97.61@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.61@KRBTEST.COM" created.
I20260620 14:50:32.431249 13921 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:50:32.409492 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:43801 (local address 127.0.0.1:49010)
0620 14:50:32.410555 (+ 1063us) negotiation.cc:107] Waiting for socket to connect
0620 14:50:32.410604 (+ 49us) client_negotiation.cc:174] Beginning negotiation
0620 14:50:32.412535 (+ 1931us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 14:50:32.413582 (+ 1047us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 14:50:32.413589 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 14:50:32.414346 (+ 757us) client_negotiation.cc:189] Negotiated authn=SASL
0620 14:50:32.414697 (+ 351us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:50:32.414717 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:50:32.416455 (+ 1738us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:50:32.416461 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:50:32.417590 (+ 1129us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:50:32.417604 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:50:32.418687 (+ 1083us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:50:32.418698 (+ 11us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:50:32.418928 (+ 230us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:50:32.419698 (+ 770us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 14:50:32.419740 (+ 42us) client_negotiation.cc:656] calling sasl_client_start()
0620 14:50:32.422939 (+ 3199us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 14:50:32.426707 (+ 3768us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 14:50:32.426720 (+ 13us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 14:50:32.426734 (+ 14us) client_negotiation.cc:760] calling sasl_client_step()
0620 14:50:32.427177 (+ 443us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 14:50:32.427641 (+ 464us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 14:50:32.427647 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 14:50:32.427650 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0620 14:50:32.427803 (+ 153us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 14:50:32.428755 (+ 952us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 14:50:32.428765 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 14:50:32.429021 (+ 256us) client_negotiation.cc:769] Sending connection context
0620 14:50:32.429279 (+ 258us) client_negotiation.cc:240] Negotiation successful
0620 14:50:32.429829 (+ 550us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":756,"spinlock_wait_cycles":65408,"thread_start_us":517,"threads_started":1}
I20260620 14:50:32.432304 13920 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:50:32.409492 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49010 (local address 127.0.97.62:43801)
0620 14:50:32.410480 (+ 988us) server_negotiation.cc:206] Beginning negotiation
0620 14:50:32.410504 (+ 24us) server_negotiation.cc:399] Waiting for connection header
0620 14:50:32.410730 (+ 226us) server_negotiation.cc:407] Connection header received
0620 14:50:32.412995 (+ 2265us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:50:32.413006 (+ 11us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:50:32.413342 (+ 336us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:50:32.413567 (+ 225us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:50:32.414914 (+ 1347us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:50:32.416299 (+ 1385us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:50:32.417751 (+ 1452us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:50:32.418348 (+ 597us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:50:32.418589 (+ 241us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:50:32.423213 (+ 4624us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 14:50:32.423222 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 14:50:32.423236 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 14:50:32.423296 (+ 60us) server_negotiation.cc:922] calling sasl_server_start()
0620 14:50:32.426546 (+ 3250us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:50:32.427337 (+ 791us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:50:32.427345 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:50:32.427347 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:50:32.427499 (+ 152us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:50:32.427912 (+ 413us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:50:32.427920 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:50:32.427923 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:50:32.428479 (+ 556us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 14:50:32.428701 (+ 222us) server_negotiation.cc:1035] Waiting for connection context
0620 14:50:32.431710 (+ 3009us) server_negotiation.cc:299] Negotiation successful
0620 14:50:32.431974 (+ 264us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":727,"thread_start_us":425,"threads_started":1}
I20260620 14:50:32.435206 13917 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.97.61" port: 36315 } has no permanent_uuid. Determining permanent_uuid...
I20260620 14:50:32.436407 13921 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:50:32.435898 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.61:36315 (local address 127.0.0.1:60638)
0620 14:50:32.436105 (+ 207us) negotiation.cc:107] Waiting for socket to connect
0620 14:50:32.436249 (+ 144us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.61:36315: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":58}
W20260620 14:50:32.436682 13866 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.97.61:36315: connect: Connection refused (error 111)
I20260620 14:50:32.438486 13921 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:50:32.437969 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.61:36315 (local address 127.0.0.1:60650)
0620 14:50:32.438194 (+ 225us) negotiation.cc:107] Waiting for socket to connect
0620 14:50:32.438297 (+ 103us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.61:36315: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":96}
W20260620 14:50:32.438750 13917 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.97.61:36315: Network error: Client connection negotiation failed: client connection to 127.0.97.61:36315: connect: Connection refused (error 111)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.61 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskUci3DU/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781966829091300-388-0/krb5kdc/kudu_127.0.97.61.keytab.
Entry for principal kudu/127.0.97.61 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskUci3DU/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781966829091300-388-0/krb5kdc/kudu_127.0.97.61.keytab.
I20260620 14:50:32.452631 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskUci3DU/build/asan/bin/kudu
/tmp/dist-test-taskUci3DU/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskUci3DU/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781966829091300-388-0/minicluster-data/master-1/wal
--fs_data_dirs=/tmp/dist-test-taskUci3DU/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781966829091300-388-0/minicluster-data/master-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskUci3DU/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781966829091300-388-0/minicluster-data/master-1/logs
--server_dump_info_path=/tmp/dist-test-taskUci3DU/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781966829091300-388-0/minicluster-data/master-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.61:36315
--webserver_interface=127.0.97.61
--webserver_port=0
--keytab_file=/tmp/dist-test-taskUci3DU/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781966829091300-388-0/krb5kdc/kudu_127.0.97.61.keytab
--principal=kudu/127.0.97.61
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:34493
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.97.62:43801,127.0.97.61:36315,127.0.97.60:45215
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskUci3DU/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781966829091300-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskUci3DU/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781966829091300-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskUci3DU/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781966829091300-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260620 14:50:32.488207 13917 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.97.61" port: 36315 } attempt: 1
I20260620 14:50:32.489502 13921 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:50:32.488952 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.61:36315 (local address 127.0.0.1:60656)
0620 14:50:32.489198 (+ 246us) negotiation.cc:107] Waiting for socket to connect
0620 14:50:32.489336 (+ 138us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.61:36315: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":83}
I20260620 14:50:32.490590 13921 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:50:32.490182 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.61:36315 (local address 127.0.0.1:60662)
0620 14:50:32.490336 (+ 154us) negotiation.cc:107] Waiting for socket to connect
0620 14:50:32.490424 (+ 88us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.61:36315: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":48}
W20260620 14:50:32.490792 13917 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.97.61:36315: Network error: Client connection negotiation failed: client connection to 127.0.97.61:36315: connect: Connection refused (error 111)
I20260620 14:50:32.559094 13917 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.97.61" port: 36315 } attempt: 2
I20260620 14:50:32.560491 13921 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:50:32.559860 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.61:36315 (local address 127.0.0.1:60664)
0620 14:50:32.560186 (+ 326us) negotiation.cc:107] Waiting for socket to connect
0620 14:50:32.560324 (+ 138us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.61:36315: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":87}
I20260620 14:50:32.561582 13921 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:50:32.561148 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.61:36315 (local address 127.0.0.1:60668)
0620 14:50:32.561299 (+ 151us) negotiation.cc:107] Waiting for socket to connect
0620 14:50:32.561398 (+ 99us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.61:36315: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":49}
W20260620 14:50:32.561802 13917 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.97.61:36315: Network error: Client connection negotiation failed: client connection to 127.0.97.61:36315: connect: Connection refused (error 111)
I20260620 14:50:32.653141 13917 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.97.61" port: 36315 } attempt: 3
I20260620 14:50:32.654613 13921 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:50:32.653932 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.61:36315 (local address 127.0.0.1:60670)
0620 14:50:32.654228 (+ 296us) negotiation.cc:107] Waiting for socket to connect
0620 14:50:32.654376 (+ 148us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.61:36315: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":97}
I20260620 14:50:32.655949 13921 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:50:32.655389 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.61:36315 (local address 127.0.0.1:60678)
0620 14:50:32.655586 (+ 197us) negotiation.cc:107] Waiting for socket to connect
0620 14:50:32.655740 (+ 154us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.61:36315: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":55}
W20260620 14:50:32.656190 13917 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.97.61:36315: Network error: Client connection negotiation failed: client connection to 127.0.97.61:36315: connect: Connection refused (error 111)
W20260620 14:50:32.799218 13924 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 14:50:32.799520 13924 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
I20260620 14:50:32.799458 13917 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.97.61" port: 36315 } attempt: 4
W20260620 14:50:32.799708 13924 flags.cc:425] Enabled unsafe flag: --never_fsync=true
I20260620 14:50:32.800801 13921 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:50:32.800183 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.61:36315 (local address 127.0.0.1:60680)
0620 14:50:32.800530 (+ 347us) negotiation.cc:107] Waiting for socket to connect
0620 14:50:32.800641 (+ 111us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.61:36315: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":150}
I20260620 14:50:32.801898 13921 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:50:32.801469 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.61:36315 (local address 127.0.0.1:60690)
0620 14:50:32.801627 (+ 158us) negotiation.cc:107] Waiting for socket to connect
0620 14:50:32.801740 (+ 113us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.61:36315: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":47}
W20260620 14:50:32.802102 13917 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.97.61:36315: Network error: Client connection negotiation failed: client connection to 127.0.97.61:36315: connect: Connection refused (error 111)
W20260620 14:50:32.808046 13924 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 14:50:32.808161 13924 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 14:50:32.808209 13924 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 14:50:32.808267 13924 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 14:50:32.808310 13924 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 14:50:32.818903 13924 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:34493
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskUci3DU/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781966829091300-388-0/minicluster-data/master-1/data
--fs_wal_dir=/tmp/dist-test-taskUci3DU/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781966829091300-388-0/minicluster-data/master-1/wal
--ipki_ca_key_size=768
--master_addresses=127.0.97.62:43801,127.0.97.61:36315,127.0.97.60:45215
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskUci3DU/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781966829091300-388-0/krb5kdc/kudu_127.0.97.61.keytab
--principal=kudu/127.0.97.61
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.61:36315
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskUci3DU/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781966829091300-388-0/minicluster-data/master-1/data/info.pb
--webserver_interface=127.0.97.61
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskUci3DU/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781966829091300-388-0/minicluster-data/master-1/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 13:41:20 UTC on ccycloud.quasar-vdagoi.root.comops.site
build id 4457462
ASAN enabled
I20260620 14:50:32.820348 13924 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 14:50:32.821718 13924 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 14:50:32.833935 13929 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 14:50:32.833956 13932 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 14:50:32.833916 13930 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 14:50:32.834877 13924 server_base.cc:1034] running on GCE node
I20260620 14:50:32.835943 13924 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 14:50:32.837414 13924 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 14:50:32.838614 13924 hybrid_clock.cc:648] HybridClock initialized: now 1781967032838567 us; error 38 us; skew 500 ppm
I20260620 14:50:32.838989 13924 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 14:50:32 dist-test-slave-q9j6 krb5kdc[13837](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781967032, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.61@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 14:50:32.842141 13924 init.cc:377] Logged in from keytab as kudu/127.0.97.61@KRBTEST.COM (short username kudu)
I20260620 14:50:32.843832 13924 webserver.cc:460] Webserver started at http://127.0.97.61:44185/ using document root <none> and password file <none>
I20260620 14:50:32.844558 13924 fs_manager.cc:359] Metadata directory not provided
I20260620 14:50:32.844713 13924 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 14:50:32.845023 13924 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 14:50:32.848147 13924 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskUci3DU/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781966829091300-388-0/minicluster-data/master-1/data/instance:
uuid: "915d2b26a77b4d28a0885933db7c65e7"
format_stamp: "Formatted at 2026-06-20 14:50:32 on dist-test-slave-q9j6"
server_key: "ff99d1e100a62ecd7dc29609706cb4f6"
server_key_iv: "696c3eb4b7132d46644bc892554f7282"
server_key_version: "encryptionkey@0"
I20260620 14:50:32.848919 13924 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskUci3DU/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781966829091300-388-0/minicluster-data/master-1/wal/instance:
uuid: "915d2b26a77b4d28a0885933db7c65e7"
format_stamp: "Formatted at 2026-06-20 14:50:32 on dist-test-slave-q9j6"
server_key: "ff99d1e100a62ecd7dc29609706cb4f6"
server_key_iv: "696c3eb4b7132d46644bc892554f7282"
server_key_version: "encryptionkey@0"
I20260620 14:50:32.854270 13924 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260620 14:50:32.857959 13939 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 14:50:32.859404 13924 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.000s sys 0.002s
I20260620 14:50:32.859648 13924 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskUci3DU/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781966829091300-388-0/minicluster-data/master-1/data,/tmp/dist-test-taskUci3DU/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781966829091300-388-0/minicluster-data/master-1/wal
uuid: "915d2b26a77b4d28a0885933db7c65e7"
format_stamp: "Formatted at 2026-06-20 14:50:32 on dist-test-slave-q9j6"
server_key: "ff99d1e100a62ecd7dc29609706cb4f6"
server_key_iv: "696c3eb4b7132d46644bc892554f7282"
server_key_version: "encryptionkey@0"
I20260620 14:50:32.859870 13924 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskUci3DU/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781966829091300-388-0/minicluster-data/master-1/wal
metadata directory: /tmp/dist-test-taskUci3DU/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781966829091300-388-0/minicluster-data/master-1/wal
1 data directories: /tmp/dist-test-taskUci3DU/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781966829091300-388-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 14:50:32.887388 13924 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 14:50:32.891656 13924 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 14:50:32.891979 13924 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 14:50:32.911303 13924 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.61:36315
I20260620 14:50:32.911326 13990 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.61:36315 every 8 connection(s)
I20260620 14:50:32.912860 13924 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskUci3DU/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781966829091300-388-0/minicluster-data/master-1/data/info.pb
I20260620 14:50:32.916412 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskUci3DU/build/asan/bin/kudu as pid 13924
I20260620 14:50:32.916625 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskUci3DU/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781966829091300-388-0/minicluster-data/master-1/wal/instance
I20260620 14:50:32.916967 13991 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 14:50:32.917124 388 external_mini_cluster.cc:1428] Setting key d5b3fbcb2a8c04e757e8bc235a469edc
I20260620 14:50:32.923898 13991 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.97.62" port: 43801 } has no permanent_uuid. Determining permanent_uuid...
WARNING: no policy specified for kudu/127.0.97.60@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.60@KRBTEST.COM" created.
Jun 20 14:50:32 dist-test-slave-q9j6 krb5kdc[13837](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781967032, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.61@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260620 14:50:32.949240 13920 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:50:32.926783 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49022 (local address 127.0.97.62:43801)
0620 14:50:32.927034 (+ 251us) server_negotiation.cc:206] Beginning negotiation
0620 14:50:32.927044 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0620 14:50:32.928375 (+ 1331us) server_negotiation.cc:407] Connection header received
0620 14:50:32.930356 (+ 1981us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:50:32.930366 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:50:32.930476 (+ 110us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:50:32.930607 (+ 131us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:50:32.932526 (+ 1919us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:50:32.933761 (+ 1235us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:50:32.935451 (+ 1690us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:50:32.936256 (+ 805us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:50:32.936430 (+ 174us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:50:32.941043 (+ 4613us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 14:50:32.941054 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 14:50:32.941062 (+ 8us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 14:50:32.941122 (+ 60us) server_negotiation.cc:922] calling sasl_server_start()
0620 14:50:32.945504 (+ 4382us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:50:32.946415 (+ 911us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:50:32.946425 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:50:32.946432 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:50:32.946608 (+ 176us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:50:32.947047 (+ 439us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:50:32.947056 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:50:32.947063 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:50:32.947532 (+ 469us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 14:50:32.947789 (+ 257us) server_negotiation.cc:1035] Waiting for connection context
0620 14:50:32.948791 (+ 1002us) server_negotiation.cc:299] Negotiation successful
0620 14:50:32.948987 (+ 196us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":77}
I20260620 14:50:32.950811 13994 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:50:32.927241 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:43801 (local address 127.0.0.1:49022)
0620 14:50:32.928158 (+ 917us) negotiation.cc:107] Waiting for socket to connect
0620 14:50:32.928226 (+ 68us) client_negotiation.cc:174] Beginning negotiation
0620 14:50:32.930030 (+ 1804us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 14:50:32.930885 (+ 855us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 14:50:32.930894 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 14:50:32.931465 (+ 571us) client_negotiation.cc:189] Negotiated authn=SASL
0620 14:50:32.932299 (+ 834us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:50:32.932320 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:50:32.933974 (+ 1654us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:50:32.933978 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:50:32.935272 (+ 1294us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:50:32.935286 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:50:32.936491 (+ 1205us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:50:32.936496 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:50:32.936722 (+ 226us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:50:32.937520 (+ 798us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 14:50:32.937544 (+ 24us) client_negotiation.cc:656] calling sasl_client_start()
0620 14:50:32.940724 (+ 3180us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 14:50:32.945709 (+ 4985us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 14:50:32.945718 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 14:50:32.945734 (+ 16us) client_negotiation.cc:760] calling sasl_client_step()
0620 14:50:32.946253 (+ 519us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 14:50:32.946740 (+ 487us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 14:50:32.946743 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 14:50:32.946746 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0620 14:50:32.946925 (+ 179us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 14:50:32.947775 (+ 850us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 14:50:32.947786 (+ 11us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 14:50:32.948304 (+ 518us) client_negotiation.cc:769] Sending connection context
0620 14:50:32.948655 (+ 351us) client_negotiation.cc:240] Negotiation successful
0620 14:50:32.949264 (+ 609us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":620,"thread_start_us":419,"threads_started":1}
I20260620 14:50:32.952715 13991 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.97.61" port: 36315 } has no permanent_uuid. Determining permanent_uuid...
Jun 20 14:50:32 dist-test-slave-q9j6 krb5kdc[13837](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781967032, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.61@KRBTEST.COM for kudu/127.0.97.61@KRBTEST.COM
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.60 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskUci3DU/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781966829091300-388-0/krb5kdc/kudu_127.0.97.60.keytab.
Entry for principal kudu/127.0.97.60 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskUci3DU/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781966829091300-388-0/krb5kdc/kudu_127.0.97.60.keytab.
I20260620 14:50:32.970319 13994 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:50:32.953485 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.61:36315 (local address 127.0.0.1:60696)
0620 14:50:32.953692 (+ 207us) negotiation.cc:107] Waiting for socket to connect
0620 14:50:32.953714 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0620 14:50:32.953829 (+ 115us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 14:50:32.954932 (+ 1103us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 14:50:32.954940 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 14:50:32.955449 (+ 509us) client_negotiation.cc:189] Negotiated authn=SASL
0620 14:50:32.955716 (+ 267us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:50:32.955732 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:50:32.957205 (+ 1473us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:50:32.957214 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:50:32.958791 (+ 1577us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:50:32.958808 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:50:32.959760 (+ 952us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:50:32.959770 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:50:32.960048 (+ 278us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:50:32.960716 (+ 668us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 14:50:32.960741 (+ 25us) client_negotiation.cc:656] calling sasl_client_start()
0620 14:50:32.963466 (+ 2725us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 14:50:32.967713 (+ 4247us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 14:50:32.967717 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 14:50:32.967720 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0620 14:50:32.968091 (+ 371us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 14:50:32.968479 (+ 388us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 14:50:32.968482 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 14:50:32.968484 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 14:50:32.968668 (+ 184us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 14:50:32.969339 (+ 671us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 14:50:32.969342 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 14:50:32.969487 (+ 145us) client_negotiation.cc:769] Sending connection context
0620 14:50:32.969606 (+ 119us) client_negotiation.cc:240] Negotiation successful
0620 14:50:32.969849 (+ 243us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":52}
I20260620 14:50:32.970525 13996 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:50:32.953728 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60696 (local address 127.0.97.61:36315)
0620 14:50:32.954462 (+ 734us) server_negotiation.cc:206] Beginning negotiation
0620 14:50:32.954469 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0620 14:50:32.954496 (+ 27us) server_negotiation.cc:407] Connection header received
0620 14:50:32.954550 (+ 54us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:50:32.954557 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:50:32.954762 (+ 205us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:50:32.954893 (+ 131us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:50:32.955887 (+ 994us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:50:32.957032 (+ 1145us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:50:32.958933 (+ 1901us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:50:32.959528 (+ 595us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:50:32.959724 (+ 196us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:50:32.963868 (+ 4144us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 14:50:32.963887 (+ 19us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 14:50:32.963908 (+ 21us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 14:50:32.963955 (+ 47us) server_negotiation.cc:922] calling sasl_server_start()
0620 14:50:32.967513 (+ 3558us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:50:32.968238 (+ 725us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:50:32.968245 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:50:32.968247 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:50:32.968371 (+ 124us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:50:32.968772 (+ 401us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:50:32.968775 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:50:32.968778 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:50:32.969215 (+ 437us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 14:50:32.969456 (+ 241us) server_negotiation.cc:1035] Waiting for connection context
0620 14:50:32.969804 (+ 348us) server_negotiation.cc:299] Negotiation successful
0620 14:50:32.970076 (+ 272us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":576,"thread_start_us":446,"threads_started":1}
I20260620 14:50:32.972540 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskUci3DU/build/asan/bin/kudu
/tmp/dist-test-taskUci3DU/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskUci3DU/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781966829091300-388-0/minicluster-data/master-2/wal
--fs_data_dirs=/tmp/dist-test-taskUci3DU/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781966829091300-388-0/minicluster-data/master-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskUci3DU/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781966829091300-388-0/minicluster-data/master-2/logs
--server_dump_info_path=/tmp/dist-test-taskUci3DU/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781966829091300-388-0/minicluster-data/master-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.60:45215
--webserver_interface=127.0.97.60
--webserver_port=0
--keytab_file=/tmp/dist-test-taskUci3DU/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781966829091300-388-0/krb5kdc/kudu_127.0.97.60.keytab
--principal=kudu/127.0.97.60
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:34493
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.97.62:43801,127.0.97.61:36315,127.0.97.60:45215
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskUci3DU/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781966829091300-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskUci3DU/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781966829091300-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskUci3DU/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781966829091300-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260620 14:50:32.973348 13991 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.97.60" port: 45215 } has no permanent_uuid. Determining permanent_uuid...
I20260620 14:50:32.974727 13994 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:50:32.974071 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.60:45215 (local address 127.0.0.1:49752)
0620 14:50:32.974310 (+ 239us) negotiation.cc:107] Waiting for socket to connect
0620 14:50:32.974476 (+ 166us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.60:45215: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":70}
W20260620 14:50:32.975028 13940 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.97.60:45215: connect: Connection refused (error 111)
I20260620 14:50:32.976790 13994 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:50:32.976282 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.60:45215 (local address 127.0.0.1:49754)
0620 14:50:32.976486 (+ 204us) negotiation.cc:107] Waiting for socket to connect
0620 14:50:32.976608 (+ 122us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.60:45215: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":53}
W20260620 14:50:32.977059 13991 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.97.60:45215: Network error: Client connection negotiation failed: client connection to 127.0.97.60:45215: connect: Connection refused (error 111)
I20260620 14:50:33.026419 13991 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.97.60" port: 45215 } attempt: 1
I20260620 14:50:33.027926 13994 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:50:33.027242 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.60:45215 (local address 127.0.0.1:49760)
0620 14:50:33.027510 (+ 268us) negotiation.cc:107] Waiting for socket to connect
0620 14:50:33.027690 (+ 180us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.60:45215: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":91}
I20260620 14:50:33.029101 13994 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:50:33.028743 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.60:45215 (local address 127.0.0.1:49772)
0620 14:50:33.028890 (+ 147us) negotiation.cc:107] Waiting for socket to connect
0620 14:50:33.028981 (+ 91us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.60:45215: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":47}
W20260620 14:50:33.029309 13991 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.97.60:45215: Network error: Client connection negotiation failed: client connection to 127.0.97.60:45215: connect: Connection refused (error 111)
I20260620 14:50:33.097546 13991 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.97.60" port: 45215 } attempt: 2
I20260620 14:50:33.098889 13994 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:50:33.098415 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.60:45215 (local address 127.0.0.1:49788)
0620 14:50:33.098632 (+ 217us) negotiation.cc:107] Waiting for socket to connect
0620 14:50:33.098726 (+ 94us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.60:45215: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":85}
I20260620 14:50:33.100128 13994 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:50:33.099720 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.60:45215 (local address 127.0.0.1:49804)
0620 14:50:33.099908 (+ 188us) negotiation.cc:107] Waiting for socket to connect
0620 14:50:33.099983 (+ 75us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.60:45215: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":48}
W20260620 14:50:33.100365 13991 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.97.60:45215: Network error: Client connection negotiation failed: client connection to 127.0.97.60:45215: connect: Connection refused (error 111)
I20260620 14:50:33.101467 13917 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.97.61" port: 36315 } attempt: 5
Jun 20 14:50:33 dist-test-slave-q9j6 krb5kdc[13837](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781967032, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.62@KRBTEST.COM for kudu/127.0.97.61@KRBTEST.COM
I20260620 14:50:33.116549 13921 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:50:33.102138 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.61:36315 (local address 127.0.0.1:60710)
0620 14:50:33.102336 (+ 198us) negotiation.cc:107] Waiting for socket to connect
0620 14:50:33.102359 (+ 23us) client_negotiation.cc:174] Beginning negotiation
0620 14:50:33.102496 (+ 137us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 14:50:33.102822 (+ 326us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 14:50:33.102825 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 14:50:33.103201 (+ 376us) client_negotiation.cc:189] Negotiated authn=SASL
0620 14:50:33.103388 (+ 187us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:50:33.103398 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:50:33.104851 (+ 1453us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:50:33.104855 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:50:33.105727 (+ 872us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:50:33.105740 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:50:33.106753 (+ 1013us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:50:33.106757 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:50:33.106927 (+ 170us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:50:33.107683 (+ 756us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 14:50:33.107703 (+ 20us) client_negotiation.cc:656] calling sasl_client_start()
0620 14:50:33.109814 (+ 2111us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 14:50:33.113818 (+ 4004us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 14:50:33.113823 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 14:50:33.113825 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 14:50:33.114260 (+ 435us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 14:50:33.114779 (+ 519us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 14:50:33.114783 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 14:50:33.114786 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0620 14:50:33.114959 (+ 173us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 14:50:33.115728 (+ 769us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 14:50:33.115732 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 14:50:33.115888 (+ 156us) client_negotiation.cc:769] Sending connection context
0620 14:50:33.116015 (+ 127us) client_negotiation.cc:240] Negotiation successful
0620 14:50:33.116292 (+ 277us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":63}
I20260620 14:50:33.116815 13996 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:50:33.102316 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60710 (local address 127.0.97.61:36315)
0620 14:50:33.102486 (+ 170us) server_negotiation.cc:206] Beginning negotiation
0620 14:50:33.102493 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0620 14:50:33.102515 (+ 22us) server_negotiation.cc:407] Connection header received
0620 14:50:33.102587 (+ 72us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:50:33.102592 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:50:33.102684 (+ 92us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:50:33.102793 (+ 109us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:50:33.103555 (+ 762us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:50:33.104693 (+ 1138us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:50:33.105880 (+ 1187us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:50:33.106590 (+ 710us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:50:33.106724 (+ 134us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:50:33.110085 (+ 3361us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 14:50:33.110092 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 14:50:33.110096 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 14:50:33.110141 (+ 45us) server_negotiation.cc:922] calling sasl_server_start()
0620 14:50:33.113586 (+ 3445us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:50:33.114428 (+ 842us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:50:33.114433 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:50:33.114437 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:50:33.114600 (+ 163us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:50:33.115109 (+ 509us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:50:33.115115 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:50:33.115118 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:50:33.115540 (+ 422us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 14:50:33.115736 (+ 196us) server_negotiation.cc:1035] Waiting for connection context
0620 14:50:33.116164 (+ 428us) server_negotiation.cc:299] Negotiation successful
0620 14:50:33.116408 (+ 244us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":57}
I20260620 14:50:33.117983 13917 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.97.60" port: 45215 } has no permanent_uuid. Determining permanent_uuid...
I20260620 14:50:33.119012 13921 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:50:33.118615 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.60:45215 (local address 127.0.0.1:49812)
0620 14:50:33.118784 (+ 169us) negotiation.cc:107] Waiting for socket to connect
0620 14:50:33.118868 (+ 84us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.60:45215: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":55}
I20260620 14:50:33.119930 13921 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:50:33.119567 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.60:45215 (local address 127.0.0.1:49828)
0620 14:50:33.119762 (+ 195us) negotiation.cc:107] Waiting for socket to connect
0620 14:50:33.119831 (+ 69us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.60:45215: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":81}
W20260620 14:50:33.120180 13917 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.97.60:45215: Network error: Client connection negotiation failed: client connection to 127.0.97.60:45215: connect: Connection refused (error 111)
I20260620 14:50:33.171445 13917 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.97.60" port: 45215 } attempt: 1
I20260620 14:50:33.173013 13921 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:50:33.172371 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.60:45215 (local address 127.0.0.1:49832)
0620 14:50:33.172615 (+ 244us) negotiation.cc:107] Waiting for socket to connect
0620 14:50:33.172757 (+ 142us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.60:45215: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":71}
I20260620 14:50:33.174324 13921 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:50:33.173787 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.60:45215 (local address 127.0.0.1:49842)
0620 14:50:33.174005 (+ 218us) negotiation.cc:107] Waiting for socket to connect
0620 14:50:33.174141 (+ 136us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.60:45215: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":67}
W20260620 14:50:33.174578 13917 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.97.60:45215: Network error: Client connection negotiation failed: client connection to 127.0.97.60:45215: connect: Connection refused (error 111)
I20260620 14:50:33.191654 13991 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.97.60" port: 45215 } attempt: 3
I20260620 14:50:33.192830 13994 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:50:33.192336 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.60:45215 (local address 127.0.0.1:49846)
0620 14:50:33.192573 (+ 237us) negotiation.cc:107] Waiting for socket to connect
0620 14:50:33.192665 (+ 92us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.60:45215: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":69}
I20260620 14:50:33.194104 13994 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:50:33.193645 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.60:45215 (local address 127.0.0.1:49854)
0620 14:50:33.193896 (+ 251us) negotiation.cc:107] Waiting for socket to connect
0620 14:50:33.193976 (+ 80us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.60:45215: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":143}
W20260620 14:50:33.194324 13991 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.97.60:45215: Network error: Client connection negotiation failed: client connection to 127.0.97.60:45215: connect: Connection refused (error 111)
I20260620 14:50:33.242826 13917 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.97.60" port: 45215 } attempt: 2
I20260620 14:50:33.244083 13921 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:50:33.243542 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.60:45215 (local address 127.0.0.1:49870)
0620 14:50:33.243783 (+ 241us) negotiation.cc:107] Waiting for socket to connect
0620 14:50:33.243877 (+ 94us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.60:45215: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":93}
I20260620 14:50:33.245086 13921 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:50:33.244744 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.60:45215 (local address 127.0.0.1:49886)
0620 14:50:33.244888 (+ 144us) negotiation.cc:107] Waiting for socket to connect
0620 14:50:33.244966 (+ 78us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.60:45215: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":48}
W20260620 14:50:33.245285 13917 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.97.60:45215: Network error: Client connection negotiation failed: client connection to 127.0.97.60:45215: connect: Connection refused (error 111)
W20260620 14:50:33.320811 13998 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 14:50:33.321193 13998 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 14:50:33.321343 13998 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 14:50:33.328949 13998 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 14:50:33.329100 13998 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 14:50:33.329213 13998 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 14:50:33.329305 13998 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 14:50:33.329383 13998 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 14:50:33.337563 13991 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.97.60" port: 45215 } attempt: 4
I20260620 14:50:33.338734 13994 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:50:33.338233 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.60:45215 (local address 127.0.0.1:49888)
0620 14:50:33.338481 (+ 248us) negotiation.cc:107] Waiting for socket to connect
0620 14:50:33.338574 (+ 93us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.60:45215: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":70}
I20260620 14:50:33.339814 13994 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:50:33.339425 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.60:45215 (local address 127.0.0.1:49900)
0620 14:50:33.339582 (+ 157us) negotiation.cc:107] Waiting for socket to connect
0620 14:50:33.339710 (+ 128us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.60:45215: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":54}
W20260620 14:50:33.340039 13991 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.97.60:45215: Network error: Client connection negotiation failed: client connection to 127.0.97.60:45215: connect: Connection refused (error 111)
I20260620 14:50:33.340130 13998 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:34493
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskUci3DU/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781966829091300-388-0/minicluster-data/master-2/data
--fs_wal_dir=/tmp/dist-test-taskUci3DU/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781966829091300-388-0/minicluster-data/master-2/wal
--ipki_ca_key_size=768
--master_addresses=127.0.97.62:43801,127.0.97.61:36315,127.0.97.60:45215
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskUci3DU/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781966829091300-388-0/krb5kdc/kudu_127.0.97.60.keytab
--principal=kudu/127.0.97.60
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.60:45215
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskUci3DU/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781966829091300-388-0/minicluster-data/master-2/data/info.pb
--webserver_interface=127.0.97.60
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskUci3DU/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781966829091300-388-0/minicluster-data/master-2/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 13:41:20 UTC on ccycloud.quasar-vdagoi.root.comops.site
build id 4457462
ASAN enabled
I20260620 14:50:33.341538 13998 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 14:50:33.342908 13998 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260620 14:50:33.351485 13917 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.97.60" port: 45215 } attempt: 3
I20260620 14:50:33.352605 13921 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:50:33.352123 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.60:45215 (local address 127.0.0.1:49914)
0620 14:50:33.352352 (+ 229us) negotiation.cc:107] Waiting for socket to connect
0620 14:50:33.352445 (+ 93us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.60:45215: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":62}
I20260620 14:50:33.353837 13921 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:50:33.353357 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.60:45215 (local address 127.0.0.1:49928)
0620 14:50:33.353567 (+ 210us) negotiation.cc:107] Waiting for socket to connect
0620 14:50:33.353678 (+ 111us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.60:45215: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":63}
W20260620 14:50:33.354130 13917 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.97.60:45215: Network error: Client connection negotiation failed: client connection to 127.0.97.60:45215: connect: Connection refused (error 111)
W20260620 14:50:33.355188 14004 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 14:50:33.355300 14006 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 14:50:33.355244 14003 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 14:50:33.356741 13998 server_base.cc:1034] running on GCE node
I20260620 14:50:33.357709 13998 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 14:50:33.359191 13998 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 14:50:33.360419 13998 hybrid_clock.cc:648] HybridClock initialized: now 1781967033360371 us; error 43 us; skew 500 ppm
I20260620 14:50:33.360793 13998 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 14:50:33 dist-test-slave-q9j6 krb5kdc[13837](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781967033, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.60@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 14:50:33.364068 13998 init.cc:377] Logged in from keytab as kudu/127.0.97.60@KRBTEST.COM (short username kudu)
I20260620 14:50:33.365676 13998 webserver.cc:460] Webserver started at http://127.0.97.60:45751/ using document root <none> and password file <none>
I20260620 14:50:33.366410 13998 fs_manager.cc:359] Metadata directory not provided
I20260620 14:50:33.366554 13998 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 14:50:33.366852 13998 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 14:50:33.369975 13998 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskUci3DU/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781966829091300-388-0/minicluster-data/master-2/data/instance:
uuid: "d9b9661d8c674b9cbde85494cdd8f84b"
format_stamp: "Formatted at 2026-06-20 14:50:33 on dist-test-slave-q9j6"
server_key: "f103b6c985bc8335a764e8db309e294d"
server_key_iv: "5c3c38a83ebb0f76a6e9cda7768bf909"
server_key_version: "encryptionkey@0"
I20260620 14:50:33.370798 13998 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskUci3DU/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781966829091300-388-0/minicluster-data/master-2/wal/instance:
uuid: "d9b9661d8c674b9cbde85494cdd8f84b"
format_stamp: "Formatted at 2026-06-20 14:50:33 on dist-test-slave-q9j6"
server_key: "f103b6c985bc8335a764e8db309e294d"
server_key_iv: "5c3c38a83ebb0f76a6e9cda7768bf909"
server_key_version: "encryptionkey@0"
I20260620 14:50:33.376492 13998 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.001s
I20260620 14:50:33.380112 14013 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 14:50:33.381565 13998 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.003s
I20260620 14:50:33.381760 13998 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskUci3DU/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781966829091300-388-0/minicluster-data/master-2/data,/tmp/dist-test-taskUci3DU/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781966829091300-388-0/minicluster-data/master-2/wal
uuid: "d9b9661d8c674b9cbde85494cdd8f84b"
format_stamp: "Formatted at 2026-06-20 14:50:33 on dist-test-slave-q9j6"
server_key: "f103b6c985bc8335a764e8db309e294d"
server_key_iv: "5c3c38a83ebb0f76a6e9cda7768bf909"
server_key_version: "encryptionkey@0"
I20260620 14:50:33.381974 13998 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskUci3DU/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781966829091300-388-0/minicluster-data/master-2/wal
metadata directory: /tmp/dist-test-taskUci3DU/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781966829091300-388-0/minicluster-data/master-2/wal
1 data directories: /tmp/dist-test-taskUci3DU/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781966829091300-388-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 14:50:33.400701 13998 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 14:50:33.405493 13998 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 14:50:33.405836 13998 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 14:50:33.424998 13998 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.60:45215
I20260620 14:50:33.425021 14064 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.60:45215 every 8 connection(s)
I20260620 14:50:33.426477 13998 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskUci3DU/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781966829091300-388-0/minicluster-data/master-2/data/info.pb
I20260620 14:50:33.430480 14065 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 14:50:33.436678 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskUci3DU/build/asan/bin/kudu as pid 13998
I20260620 14:50:33.436889 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskUci3DU/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781966829091300-388-0/minicluster-data/master-2/wal/instance
I20260620 14:50:33.437425 388 external_mini_cluster.cc:1428] Setting key db299ce3af96a91f8d4ec2f11ab40367
I20260620 14:50:33.437548 14065 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.97.62" port: 43801 } has no permanent_uuid. Determining permanent_uuid...
Jun 20 14:50:33 dist-test-slave-q9j6 krb5kdc[13837](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781967033, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.60@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
Jun 20 14:50:33 dist-test-slave-q9j6 krb5kdc[13837](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781967031, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.60@KRBTEST.COM
I20260620 14:50:33.459661 13920 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:50:33.440754 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49036 (local address 127.0.97.62:43801)
0620 14:50:33.440969 (+ 215us) server_negotiation.cc:206] Beginning negotiation
0620 14:50:33.440975 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 14:50:33.441933 (+ 958us) server_negotiation.cc:407] Connection header received
0620 14:50:33.442644 (+ 711us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:50:33.442651 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:50:33.442737 (+ 86us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:50:33.442856 (+ 119us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:50:33.444838 (+ 1982us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:50:33.446024 (+ 1186us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:50:33.448293 (+ 2269us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:50:33.448901 (+ 608us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:50:33.449044 (+ 143us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:50:33.453310 (+ 4266us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 14:50:33.453315 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 14:50:33.453318 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 14:50:33.453356 (+ 38us) server_negotiation.cc:922] calling sasl_server_start()
0620 14:50:33.456405 (+ 3049us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:50:33.457215 (+ 810us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:50:33.457219 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:50:33.457222 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:50:33.457364 (+ 142us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:50:33.457809 (+ 445us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:50:33.457813 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:50:33.457815 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:50:33.458075 (+ 260us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 14:50:33.458246 (+ 171us) server_negotiation.cc:1035] Waiting for connection context
0620 14:50:33.459021 (+ 775us) server_negotiation.cc:299] Negotiation successful
0620 14:50:33.459272 (+ 251us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":66}
I20260620 14:50:33.460714 14068 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:50:33.439380 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49938 (local address 127.0.97.60:45215)
0620 14:50:33.440190 (+ 810us) server_negotiation.cc:206] Beginning negotiation
0620 14:50:33.440201 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0620 14:50:33.440240 (+ 39us) server_negotiation.cc:407] Connection header received
0620 14:50:33.442381 (+ 2141us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:50:33.442405 (+ 24us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:50:33.442712 (+ 307us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:50:33.442918 (+ 206us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:50:33.444836 (+ 1918us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:50:33.446517 (+ 1681us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:50:33.448562 (+ 2045us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:50:33.449405 (+ 843us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:50:33.449506 (+ 101us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:50:33.453932 (+ 4426us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 14:50:33.453941 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 14:50:33.453957 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 14:50:33.453988 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0620 14:50:33.456824 (+ 2836us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:50:33.457593 (+ 769us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:50:33.457600 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:50:33.457602 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:50:33.457735 (+ 133us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:50:33.458132 (+ 397us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:50:33.458136 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:50:33.458138 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:50:33.458617 (+ 479us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 14:50:33.458929 (+ 312us) server_negotiation.cc:1035] Waiting for connection context
0620 14:50:33.459263 (+ 334us) server_negotiation.cc:299] Negotiation successful
0620 14:50:33.459562 (+ 299us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":44,"server-negotiator.queue_time_us":602,"thread_start_us":372,"threads_started":1}
I20260620 14:50:33.460750 14069 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:50:33.440780 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:43801 (local address 127.0.0.1:49036)
0620 14:50:33.441747 (+ 967us) negotiation.cc:107] Waiting for socket to connect
0620 14:50:33.441793 (+ 46us) client_negotiation.cc:174] Beginning negotiation
0620 14:50:33.442390 (+ 597us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 14:50:33.442943 (+ 553us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 14:50:33.442950 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 14:50:33.443977 (+ 1027us) client_negotiation.cc:189] Negotiated authn=SASL
0620 14:50:33.444625 (+ 648us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:50:33.444655 (+ 30us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:50:33.446215 (+ 1560us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:50:33.446219 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:50:33.448114 (+ 1895us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:50:33.448137 (+ 23us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:50:33.449145 (+ 1008us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:50:33.449151 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:50:33.449328 (+ 177us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:50:33.450168 (+ 840us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 14:50:33.450194 (+ 26us) client_negotiation.cc:656] calling sasl_client_start()
0620 14:50:33.453081 (+ 2887us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 14:50:33.456576 (+ 3495us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 14:50:33.456597 (+ 21us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 14:50:33.456611 (+ 14us) client_negotiation.cc:760] calling sasl_client_step()
0620 14:50:33.457064 (+ 453us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 14:50:33.457477 (+ 413us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 14:50:33.457483 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 14:50:33.457487 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0620 14:50:33.457684 (+ 197us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 14:50:33.458210 (+ 526us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 14:50:33.458219 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 14:50:33.458656 (+ 437us) client_negotiation.cc:769] Sending connection context
0620 14:50:33.458914 (+ 258us) client_negotiation.cc:240] Negotiation successful
0620 14:50:33.459434 (+ 520us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":685,"mutex_wait_us":3,"spinlock_wait_cycles":17280,"thread_start_us":502,"threads_started":1}
I20260620 14:50:33.462324 14065 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.97.61" port: 36315 } has no permanent_uuid. Determining permanent_uuid...
Jun 20 14:50:33 dist-test-slave-q9j6 krb5kdc[13837](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781967033, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.60@KRBTEST.COM for kudu/127.0.97.61@KRBTEST.COM
Jun 20 14:50:33 dist-test-slave-q9j6 krb5kdc[13837](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781967031, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260620 14:50:33.477151 14069 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:50:33.463166 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.61:36315 (local address 127.0.0.1:60722)
0620 14:50:33.463322 (+ 156us) negotiation.cc:107] Waiting for socket to connect
0620 14:50:33.463343 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0620 14:50:33.463472 (+ 129us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 14:50:33.463847 (+ 375us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 14:50:33.463855 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 14:50:33.464335 (+ 480us) client_negotiation.cc:189] Negotiated authn=SASL
0620 14:50:33.464537 (+ 202us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:50:33.464549 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:50:33.465911 (+ 1362us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:50:33.465915 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:50:33.467161 (+ 1246us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:50:33.467177 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:50:33.468035 (+ 858us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:50:33.468040 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:50:33.468204 (+ 164us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:50:33.468815 (+ 611us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 14:50:33.468832 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0620 14:50:33.471237 (+ 2405us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 14:50:33.474685 (+ 3448us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 14:50:33.474691 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 14:50:33.474695 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0620 14:50:33.475074 (+ 379us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 14:50:33.475469 (+ 395us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 14:50:33.475474 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 14:50:33.475478 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0620 14:50:33.475679 (+ 201us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 14:50:33.476176 (+ 497us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 14:50:33.476181 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 14:50:33.476336 (+ 155us) client_negotiation.cc:769] Sending connection context
0620 14:50:33.476478 (+ 142us) client_negotiation.cc:240] Negotiation successful
0620 14:50:33.476780 (+ 302us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":49}
I20260620 14:50:33.477182 13996 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:50:33.463093 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60722 (local address 127.0.97.61:36315)
0620 14:50:33.463305 (+ 212us) server_negotiation.cc:206] Beginning negotiation
0620 14:50:33.463309 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 14:50:33.463456 (+ 147us) server_negotiation.cc:407] Connection header received
0620 14:50:33.463626 (+ 170us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:50:33.463633 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:50:33.463708 (+ 75us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:50:33.463788 (+ 80us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:50:33.464670 (+ 882us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:50:33.465754 (+ 1084us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:50:33.467310 (+ 1556us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:50:33.467898 (+ 588us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:50:33.468034 (+ 136us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:50:33.471440 (+ 3406us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 14:50:33.471448 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 14:50:33.471456 (+ 8us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 14:50:33.471507 (+ 51us) server_negotiation.cc:922] calling sasl_server_start()
0620 14:50:33.474538 (+ 3031us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:50:33.475213 (+ 675us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:50:33.475217 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:50:33.475220 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:50:33.475348 (+ 128us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:50:33.475790 (+ 442us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:50:33.475795 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:50:33.475798 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:50:33.476054 (+ 256us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 14:50:33.476205 (+ 151us) server_negotiation.cc:1035] Waiting for connection context
0620 14:50:33.476562 (+ 357us) server_negotiation.cc:299] Negotiation successful
0620 14:50:33.476788 (+ 226us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":83}
I20260620 14:50:33.478189 14065 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.97.60" port: 45215 } has no permanent_uuid. Determining permanent_uuid...
I20260620 14:50:33.480463 13920 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:50:33.464496 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49042 (local address 127.0.97.62:43801)
0620 14:50:33.464768 (+ 272us) server_negotiation.cc:206] Beginning negotiation
0620 14:50:33.464772 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 14:50:33.464787 (+ 15us) server_negotiation.cc:407] Connection header received
0620 14:50:33.464828 (+ 41us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:50:33.464832 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:50:33.464884 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:50:33.464957 (+ 73us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:50:33.466432 (+ 1475us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:50:33.467466 (+ 1034us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:50:33.469171 (+ 1705us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:50:33.469821 (+ 650us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:50:33.469970 (+ 149us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:50:33.474679 (+ 4709us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 14:50:33.474685 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 14:50:33.474688 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 14:50:33.474721 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0620 14:50:33.477496 (+ 2775us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:50:33.478351 (+ 855us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:50:33.478360 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:50:33.478363 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:50:33.478509 (+ 146us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:50:33.478933 (+ 424us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:50:33.478938 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:50:33.478942 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:50:33.479315 (+ 373us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 14:50:33.479651 (+ 336us) server_negotiation.cc:1035] Waiting for connection context
0620 14:50:33.479918 (+ 267us) server_negotiation.cc:299] Negotiation successful
0620 14:50:33.480147 (+ 229us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":164}
Jun 20 14:50:33 dist-test-slave-q9j6 krb5kdc[13837](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781967033, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.60@KRBTEST.COM for kudu/127.0.97.60@KRBTEST.COM
Jun 20 14:50:33 dist-test-slave-q9j6 krb5kdc[13837](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781967031, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.61@KRBTEST.COM
I20260620 14:50:33.492540 14068 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:50:33.479098 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49950 (local address 127.0.97.60:45215)
0620 14:50:33.479243 (+ 145us) server_negotiation.cc:206] Beginning negotiation
0620 14:50:33.479247 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 14:50:33.479260 (+ 13us) server_negotiation.cc:407] Connection header received
0620 14:50:33.479297 (+ 37us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:50:33.479300 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:50:33.479350 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:50:33.479452 (+ 102us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:50:33.480157 (+ 705us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:50:33.481028 (+ 871us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:50:33.482501 (+ 1473us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:50:33.483057 (+ 556us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:50:33.483160 (+ 103us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:50:33.486517 (+ 3357us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 14:50:33.486524 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 14:50:33.486529 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 14:50:33.486564 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0620 14:50:33.489891 (+ 3327us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:50:33.490548 (+ 657us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:50:33.490556 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:50:33.490559 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:50:33.490721 (+ 162us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:50:33.491057 (+ 336us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:50:33.491063 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:50:33.491066 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:50:33.491416 (+ 350us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 14:50:33.491530 (+ 114us) server_negotiation.cc:1035] Waiting for connection context
0620 14:50:33.492013 (+ 483us) server_negotiation.cc:299] Negotiation successful
0620 14:50:33.492185 (+ 172us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":43}
I20260620 14:50:33.492641 14069 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:50:33.478929 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.60:45215 (local address 127.0.0.1:49950)
0620 14:50:33.479101 (+ 172us) negotiation.cc:107] Waiting for socket to connect
0620 14:50:33.479121 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0620 14:50:33.479212 (+ 91us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 14:50:33.479464 (+ 252us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 14:50:33.479468 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 14:50:33.479900 (+ 432us) client_negotiation.cc:189] Negotiated authn=SASL
0620 14:50:33.480025 (+ 125us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:50:33.480033 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:50:33.481141 (+ 1108us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:50:33.481145 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:50:33.482371 (+ 1226us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:50:33.482385 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:50:33.483222 (+ 837us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:50:33.483229 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:50:33.483466 (+ 237us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:50:33.484100 (+ 634us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 14:50:33.484122 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0620 14:50:33.486270 (+ 2148us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 14:50:33.490043 (+ 3773us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 14:50:33.490048 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 14:50:33.490051 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0620 14:50:33.490423 (+ 372us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 14:50:33.490829 (+ 406us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 14:50:33.490833 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 14:50:33.490835 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 14:50:33.490956 (+ 121us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 14:50:33.491540 (+ 584us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 14:50:33.491548 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 14:50:33.491770 (+ 222us) client_negotiation.cc:769] Sending connection context
0620 14:50:33.491936 (+ 166us) client_negotiation.cc:240] Negotiation successful
0620 14:50:33.492364 (+ 428us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":40,"mutex_wait_us":82}
I20260620 14:50:33.498344 13996 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:50:33.482215 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60738 (local address 127.0.97.61:36315)
0620 14:50:33.482354 (+ 139us) server_negotiation.cc:206] Beginning negotiation
0620 14:50:33.482358 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 14:50:33.482373 (+ 15us) server_negotiation.cc:407] Connection header received
0620 14:50:33.482501 (+ 128us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:50:33.482505 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:50:33.482579 (+ 74us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:50:33.482675 (+ 96us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:50:33.484232 (+ 1557us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:50:33.484958 (+ 726us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:50:33.486556 (+ 1598us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:50:33.486981 (+ 425us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:50:33.487138 (+ 157us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:50:33.492392 (+ 5254us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 14:50:33.492399 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 14:50:33.492404 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 14:50:33.492446 (+ 42us) server_negotiation.cc:922] calling sasl_server_start()
0620 14:50:33.495695 (+ 3249us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:50:33.496377 (+ 682us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:50:33.496383 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:50:33.496387 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:50:33.496561 (+ 174us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:50:33.496877 (+ 316us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:50:33.496882 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:50:33.496885 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:50:33.497235 (+ 350us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 14:50:33.497521 (+ 286us) server_negotiation.cc:1035] Waiting for connection context
0620 14:50:33.497739 (+ 218us) server_negotiation.cc:299] Negotiation successful
0620 14:50:33.497974 (+ 235us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":37}
I20260620 14:50:33.498988 14065 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d9b9661d8c674b9cbde85494cdd8f84b: Bootstrap starting.
I20260620 14:50:33.502861 14065 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P d9b9661d8c674b9cbde85494cdd8f84b: Neither blocks nor log segments found. Creating new log.
I20260620 14:50:33.503297 388 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260620 14:50:33.504173 14065 log.cc:824] T 00000000000000000000000000000000 P d9b9661d8c674b9cbde85494cdd8f84b: Log is configured to *not* fsync() on all Append() calls
I20260620 14:50:33.507280 14065 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d9b9661d8c674b9cbde85494cdd8f84b: No bootstrap required, opened a new log
I20260620 14:50:33.522792 14065 raft_consensus.cc:348] T 00000000000000000000000000000000 P d9b9661d8c674b9cbde85494cdd8f84b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a7d13d41ab9648f5b3b5884352620c8b" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 43801 } } peers { permanent_uuid: "915d2b26a77b4d28a0885933db7c65e7" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 36315 } } peers { permanent_uuid: "d9b9661d8c674b9cbde85494cdd8f84b" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 45215 } }
I20260620 14:50:33.523454 14065 raft_consensus.cc:374] T 00000000000000000000000000000000 P d9b9661d8c674b9cbde85494cdd8f84b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 14:50:33.523710 14065 raft_consensus.cc:729] T 00000000000000000000000000000000 P d9b9661d8c674b9cbde85494cdd8f84b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d9b9661d8c674b9cbde85494cdd8f84b, State: Initialized, Role: FOLLOWER
I20260620 14:50:33.524648 14065 consensus_queue.cc:260] T 00000000000000000000000000000000 P d9b9661d8c674b9cbde85494cdd8f84b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a7d13d41ab9648f5b3b5884352620c8b" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 43801 } } peers { permanent_uuid: "915d2b26a77b4d28a0885933db7c65e7" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 36315 } } peers { permanent_uuid: "d9b9661d8c674b9cbde85494cdd8f84b" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 45215 } }
I20260620 14:50:33.530061 13996 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:50:33.508272 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60748 (local address 127.0.97.61:36315)
0620 14:50:33.508618 (+ 346us) server_negotiation.cc:206] Beginning negotiation
0620 14:50:33.508623 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 14:50:33.509121 (+ 498us) server_negotiation.cc:407] Connection header received
0620 14:50:33.509456 (+ 335us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:50:33.509465 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:50:33.509572 (+ 107us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:50:33.509679 (+ 107us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:50:33.512821 (+ 3142us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:50:33.514329 (+ 1508us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:50:33.517495 (+ 3166us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:50:33.518419 (+ 924us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:50:33.518536 (+ 117us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:50:33.522676 (+ 4140us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 14:50:33.522682 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 14:50:33.522687 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 14:50:33.522730 (+ 43us) server_negotiation.cc:922] calling sasl_server_start()
0620 14:50:33.526395 (+ 3665us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:50:33.527368 (+ 973us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:50:33.527376 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:50:33.527379 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:50:33.527548 (+ 169us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:50:33.528076 (+ 528us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:50:33.528081 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:50:33.528084 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:50:33.528482 (+ 398us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 14:50:33.528743 (+ 261us) server_negotiation.cc:1035] Waiting for connection context
0620 14:50:33.529438 (+ 695us) server_negotiation.cc:299] Negotiation successful
0620 14:50:33.529699 (+ 261us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":228}
I20260620 14:50:33.531352 13917 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.97.60" port: 45215 } attempt: 4
I20260620 14:50:33.533258 14078 sys_catalog.cc:455] T 00000000000000000000000000000000 P d9b9661d8c674b9cbde85494cdd8f84b [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a7d13d41ab9648f5b3b5884352620c8b" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 43801 } } peers { permanent_uuid: "915d2b26a77b4d28a0885933db7c65e7" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 36315 } } peers { permanent_uuid: "d9b9661d8c674b9cbde85494cdd8f84b" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 45215 } } }
I20260620 14:50:33.535928 14078 sys_catalog.cc:458] T 00000000000000000000000000000000 P d9b9661d8c674b9cbde85494cdd8f84b [sys.catalog]: This master's current role is: FOLLOWER
I20260620 14:50:33.535787 14065 sys_catalog.cc:564] T 00000000000000000000000000000000 P d9b9661d8c674b9cbde85494cdd8f84b [sys.catalog]: configured and running, proceeding with master startup.
I20260620 14:50:33.538770 13920 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:50:33.508250 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49054 (local address 127.0.97.62:43801)
0620 14:50:33.508466 (+ 216us) server_negotiation.cc:206] Beginning negotiation
0620 14:50:33.508474 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0620 14:50:33.519093 (+ 10619us) server_negotiation.cc:407] Connection header received
0620 14:50:33.519292 (+ 199us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:50:33.519300 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:50:33.519402 (+ 102us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:50:33.519502 (+ 100us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:50:33.522031 (+ 2529us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:50:33.523162 (+ 1131us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:50:33.525370 (+ 2208us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:50:33.526098 (+ 728us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:50:33.526213 (+ 115us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:50:33.529838 (+ 3625us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 14:50:33.529845 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 14:50:33.529851 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 14:50:33.529895 (+ 44us) server_negotiation.cc:922] calling sasl_server_start()
0620 14:50:33.534347 (+ 4452us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:50:33.535515 (+ 1168us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:50:33.535524 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:50:33.535530 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:50:33.535782 (+ 252us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:50:33.536363 (+ 581us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:50:33.536371 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:50:33.536377 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:50:33.536850 (+ 473us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 14:50:33.537176 (+ 326us) server_negotiation.cc:1035] Waiting for connection context
0620 14:50:33.537879 (+ 703us) server_negotiation.cc:299] Negotiation successful
0620 14:50:33.538241 (+ 362us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":57}
I20260620 14:50:33.542002 14068 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:50:33.509471 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49954 (local address 127.0.97.60:45215)
0620 14:50:33.509685 (+ 214us) server_negotiation.cc:206] Beginning negotiation
0620 14:50:33.509690 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 14:50:33.521292 (+ 11602us) server_negotiation.cc:407] Connection header received
0620 14:50:33.521554 (+ 262us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:50:33.521562 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:50:33.521644 (+ 82us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:50:33.521716 (+ 72us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:50:33.523612 (+ 1896us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:50:33.524684 (+ 1072us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:50:33.527334 (+ 2650us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:50:33.528360 (+ 1026us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:50:33.528485 (+ 125us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:50:33.533932 (+ 5447us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 14:50:33.533938 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 14:50:33.533942 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 14:50:33.533986 (+ 44us) server_negotiation.cc:922] calling sasl_server_start()
0620 14:50:33.537800 (+ 3814us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:50:33.538904 (+ 1104us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:50:33.538912 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:50:33.538919 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:50:33.539173 (+ 254us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:50:33.539740 (+ 567us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:50:33.539748 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:50:33.539754 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:50:33.540257 (+ 503us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 14:50:33.540491 (+ 234us) server_negotiation.cc:1035] Waiting for connection context
0620 14:50:33.541147 (+ 656us) server_negotiation.cc:299] Negotiation successful
0620 14:50:33.541434 (+ 287us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":60}
Jun 20 14:50:33 dist-test-slave-q9j6 krb5kdc[13837](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781967032, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.62@KRBTEST.COM for kudu/127.0.97.60@KRBTEST.COM
W20260620 14:50:33.552369 14090 catalog_manager.cc:1558] T 00000000000000000000000000000000 P d9b9661d8c674b9cbde85494cdd8f84b: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260620 14:50:33.552556 14090 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260620 14:50:33.557152 13921 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:50:33.531952 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.60:45215 (local address 127.0.0.1:49960)
0620 14:50:33.532130 (+ 178us) negotiation.cc:107] Waiting for socket to connect
0620 14:50:33.532147 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0620 14:50:33.532266 (+ 119us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 14:50:33.540729 (+ 8463us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 14:50:33.540735 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 14:50:33.541212 (+ 477us) client_negotiation.cc:189] Negotiated authn=SASL
0620 14:50:33.541409 (+ 197us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:50:33.541421 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:50:33.543218 (+ 1797us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:50:33.543228 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:50:33.544928 (+ 1700us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:50:33.544947 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:50:33.545704 (+ 757us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:50:33.545713 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:50:33.545952 (+ 239us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:50:33.546679 (+ 727us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 14:50:33.546696 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0620 14:50:33.550599 (+ 3903us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 14:50:33.554584 (+ 3985us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 14:50:33.554590 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 14:50:33.554594 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0620 14:50:33.554953 (+ 359us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 14:50:33.555345 (+ 392us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 14:50:33.555351 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 14:50:33.555355 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0620 14:50:33.555494 (+ 139us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 14:50:33.556087 (+ 593us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 14:50:33.556092 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 14:50:33.556251 (+ 159us) client_negotiation.cc:769] Sending connection context
0620 14:50:33.556448 (+ 197us) client_negotiation.cc:240] Negotiation successful
0620 14:50:33.556760 (+ 312us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":55}
I20260620 14:50:33.557219 14079 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:50:33.532316 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49960 (local address 127.0.97.60:45215)
0620 14:50:33.540399 (+ 8083us) server_negotiation.cc:206] Beginning negotiation
0620 14:50:33.540405 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 14:50:33.540427 (+ 22us) server_negotiation.cc:407] Connection header received
0620 14:50:33.540472 (+ 45us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:50:33.540476 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:50:33.540550 (+ 74us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:50:33.540669 (+ 119us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:50:33.541623 (+ 954us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:50:33.542983 (+ 1360us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:50:33.545081 (+ 2098us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:50:33.545557 (+ 476us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:50:33.545661 (+ 104us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:50:33.550838 (+ 5177us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 14:50:33.550844 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 14:50:33.550849 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 14:50:33.550892 (+ 43us) server_negotiation.cc:922] calling sasl_server_start()
0620 14:50:33.554415 (+ 3523us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:50:33.555090 (+ 675us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:50:33.555096 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:50:33.555100 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:50:33.555233 (+ 133us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:50:33.555649 (+ 416us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:50:33.555652 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:50:33.555654 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:50:33.555952 (+ 298us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 14:50:33.556121 (+ 169us) server_negotiation.cc:1035] Waiting for connection context
0620 14:50:33.556544 (+ 423us) server_negotiation.cc:299] Negotiation successful
0620 14:50:33.556760 (+ 216us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":7913,"thread_start_us":7726,"threads_started":1}
I20260620 14:50:33.549707 13998 master_runner.cc:186] Error getting master registration for 127.0.97.62:43801: OK, instance_id { permanent_uuid: "a7d13d41ab9648f5b3b5884352620c8b" instance_seqno: 1781967032387203 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260620 14:50:33.561519 13998 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260620 14:50:33.563910 13917 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a7d13d41ab9648f5b3b5884352620c8b: Bootstrap starting.
I20260620 14:50:33.568889 13917 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P a7d13d41ab9648f5b3b5884352620c8b: Neither blocks nor log segments found. Creating new log.
I20260620 14:50:33.570297 13917 log.cc:824] T 00000000000000000000000000000000 P a7d13d41ab9648f5b3b5884352620c8b: Log is configured to *not* fsync() on all Append() calls
I20260620 14:50:33.573747 13917 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a7d13d41ab9648f5b3b5884352620c8b: No bootstrap required, opened a new log
I20260620 14:50:33.586273 13917 raft_consensus.cc:348] T 00000000000000000000000000000000 P a7d13d41ab9648f5b3b5884352620c8b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a7d13d41ab9648f5b3b5884352620c8b" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 43801 } } peers { permanent_uuid: "915d2b26a77b4d28a0885933db7c65e7" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 36315 } } peers { permanent_uuid: "d9b9661d8c674b9cbde85494cdd8f84b" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 45215 } }
I20260620 14:50:33.586723 13917 raft_consensus.cc:374] T 00000000000000000000000000000000 P a7d13d41ab9648f5b3b5884352620c8b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 14:50:33.586850 13917 raft_consensus.cc:729] T 00000000000000000000000000000000 P a7d13d41ab9648f5b3b5884352620c8b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a7d13d41ab9648f5b3b5884352620c8b, State: Initialized, Role: FOLLOWER
I20260620 14:50:33.587561 13917 consensus_queue.cc:260] T 00000000000000000000000000000000 P a7d13d41ab9648f5b3b5884352620c8b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a7d13d41ab9648f5b3b5884352620c8b" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 43801 } } peers { permanent_uuid: "915d2b26a77b4d28a0885933db7c65e7" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 36315 } } peers { permanent_uuid: "d9b9661d8c674b9cbde85494cdd8f84b" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 45215 } }
I20260620 14:50:33.588554 14093 sys_catalog.cc:455] T 00000000000000000000000000000000 P a7d13d41ab9648f5b3b5884352620c8b [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a7d13d41ab9648f5b3b5884352620c8b" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 43801 } } peers { permanent_uuid: "915d2b26a77b4d28a0885933db7c65e7" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 36315 } } peers { permanent_uuid: "d9b9661d8c674b9cbde85494cdd8f84b" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 45215 } } }
I20260620 14:50:33.588923 14093 sys_catalog.cc:458] T 00000000000000000000000000000000 P a7d13d41ab9648f5b3b5884352620c8b [sys.catalog]: This master's current role is: FOLLOWER
I20260620 14:50:33.590060 13917 sys_catalog.cc:564] T 00000000000000000000000000000000 P a7d13d41ab9648f5b3b5884352620c8b [sys.catalog]: configured and running, proceeding with master startup.
W20260620 14:50:33.600545 14104 catalog_manager.cc:1558] T 00000000000000000000000000000000 P a7d13d41ab9648f5b3b5884352620c8b: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260620 14:50:33.600806 14104 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260620 14:50:33.599025 13850 master_runner.cc:186] Error getting master registration for 127.0.97.61:36315: OK, instance_id { permanent_uuid: "915d2b26a77b4d28a0885933db7c65e7" instance_seqno: 1781967032905873 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260620 14:50:33.611846 13850 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
W20260620 14:50:33.626652 13871 tablet.cc:2369] T 00000000000000000000000000000000 P a7d13d41ab9648f5b3b5884352620c8b: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260620 14:50:33.639251 13991 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.97.60" port: 45215 } attempt: 5
Jun 20 14:50:33 dist-test-slave-q9j6 krb5kdc[13837](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781967032, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.61@KRBTEST.COM for kudu/127.0.97.60@KRBTEST.COM
I20260620 14:50:33.657424 13994 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:50:33.639996 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.60:45215 (local address 127.0.0.1:49962)
0620 14:50:33.640305 (+ 309us) negotiation.cc:107] Waiting for socket to connect
0620 14:50:33.640326 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0620 14:50:33.640456 (+ 130us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 14:50:33.640881 (+ 425us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 14:50:33.640886 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 14:50:33.641448 (+ 562us) client_negotiation.cc:189] Negotiated authn=SASL
0620 14:50:33.641671 (+ 223us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:50:33.641685 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:50:33.643165 (+ 1480us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:50:33.643171 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:50:33.644720 (+ 1549us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:50:33.644739 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:50:33.645826 (+ 1087us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:50:33.645833 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:50:33.646036 (+ 203us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:50:33.646787 (+ 751us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 14:50:33.646812 (+ 25us) client_negotiation.cc:656] calling sasl_client_start()
0620 14:50:33.649758 (+ 2946us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 14:50:33.654287 (+ 4529us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 14:50:33.654293 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 14:50:33.654298 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0620 14:50:33.654881 (+ 583us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 14:50:33.655389 (+ 508us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 14:50:33.655394 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 14:50:33.655398 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0620 14:50:33.655575 (+ 177us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 14:50:33.656325 (+ 750us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 14:50:33.656330 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 14:50:33.656509 (+ 179us) client_negotiation.cc:769] Sending connection context
0620 14:50:33.656682 (+ 173us) client_negotiation.cc:240] Negotiation successful
0620 14:50:33.657015 (+ 333us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":185}
I20260620 14:50:33.657470 14079 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:50:33.640071 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49962 (local address 127.0.97.60:45215)
0620 14:50:33.640297 (+ 226us) server_negotiation.cc:206] Beginning negotiation
0620 14:50:33.640303 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 14:50:33.640407 (+ 104us) server_negotiation.cc:407] Connection header received
0620 14:50:33.640595 (+ 188us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:50:33.640602 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:50:33.640703 (+ 101us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:50:33.640816 (+ 113us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:50:33.641849 (+ 1033us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:50:33.643009 (+ 1160us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:50:33.644873 (+ 1864us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:50:33.645654 (+ 781us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:50:33.645783 (+ 129us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:50:33.650000 (+ 4217us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 14:50:33.650007 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 14:50:33.650012 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 14:50:33.650058 (+ 46us) server_negotiation.cc:922] calling sasl_server_start()
0620 14:50:33.654048 (+ 3990us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:50:33.655043 (+ 995us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:50:33.655050 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:50:33.655054 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:50:33.655242 (+ 188us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:50:33.655747 (+ 505us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:50:33.655753 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:50:33.655757 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:50:33.656188 (+ 431us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 14:50:33.656349 (+ 161us) server_negotiation.cc:1035] Waiting for connection context
0620 14:50:33.656820 (+ 471us) server_negotiation.cc:299] Negotiation successful
0620 14:50:33.657119 (+ 299us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":88}
W20260620 14:50:33.658051 14019 tablet.cc:2369] T 00000000000000000000000000000000 P d9b9661d8c674b9cbde85494cdd8f84b: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260620 14:50:33.664476 13991 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 915d2b26a77b4d28a0885933db7c65e7: Bootstrap starting.
I20260620 14:50:33.668928 13991 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 915d2b26a77b4d28a0885933db7c65e7: Neither blocks nor log segments found. Creating new log.
I20260620 14:50:33.670363 13991 log.cc:824] T 00000000000000000000000000000000 P 915d2b26a77b4d28a0885933db7c65e7: Log is configured to *not* fsync() on all Append() calls
I20260620 14:50:33.673872 13991 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 915d2b26a77b4d28a0885933db7c65e7: No bootstrap required, opened a new log
I20260620 14:50:33.685341 13991 raft_consensus.cc:348] T 00000000000000000000000000000000 P 915d2b26a77b4d28a0885933db7c65e7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a7d13d41ab9648f5b3b5884352620c8b" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 43801 } } peers { permanent_uuid: "915d2b26a77b4d28a0885933db7c65e7" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 36315 } } peers { permanent_uuid: "d9b9661d8c674b9cbde85494cdd8f84b" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 45215 } }
I20260620 14:50:33.685890 13991 raft_consensus.cc:374] T 00000000000000000000000000000000 P 915d2b26a77b4d28a0885933db7c65e7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 14:50:33.686012 13991 raft_consensus.cc:729] T 00000000000000000000000000000000 P 915d2b26a77b4d28a0885933db7c65e7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 915d2b26a77b4d28a0885933db7c65e7, State: Initialized, Role: FOLLOWER
I20260620 14:50:33.686828 13991 consensus_queue.cc:260] T 00000000000000000000000000000000 P 915d2b26a77b4d28a0885933db7c65e7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a7d13d41ab9648f5b3b5884352620c8b" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 43801 } } peers { permanent_uuid: "915d2b26a77b4d28a0885933db7c65e7" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 36315 } } peers { permanent_uuid: "d9b9661d8c674b9cbde85494cdd8f84b" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 45215 } }
I20260620 14:50:33.690240 14107 sys_catalog.cc:455] T 00000000000000000000000000000000 P 915d2b26a77b4d28a0885933db7c65e7 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a7d13d41ab9648f5b3b5884352620c8b" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 43801 } } peers { permanent_uuid: "915d2b26a77b4d28a0885933db7c65e7" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 36315 } } peers { permanent_uuid: "d9b9661d8c674b9cbde85494cdd8f84b" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 45215 } } }
I20260620 14:50:33.690783 14107 sys_catalog.cc:458] T 00000000000000000000000000000000 P 915d2b26a77b4d28a0885933db7c65e7 [sys.catalog]: This master's current role is: FOLLOWER
I20260620 14:50:33.692016 13991 sys_catalog.cc:564] T 00000000000000000000000000000000 P 915d2b26a77b4d28a0885933db7c65e7 [sys.catalog]: configured and running, proceeding with master startup.
I20260620 14:50:33.713001 13924 master_runner.cc:263] No leader master found from master 915d2b26a77b4d28a0885933db7c65e7
I20260620 14:50:33.713179 13924 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260620 14:50:33.715979 14078 raft_consensus.cc:482] T 00000000000000000000000000000000 P d9b9661d8c674b9cbde85494cdd8f84b [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260620 14:50:33.716275 14118 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 915d2b26a77b4d28a0885933db7c65e7: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260620 14:50:33.716464 14118 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260620 14:50:33.716324 14078 raft_consensus.cc:504] T 00000000000000000000000000000000 P d9b9661d8c674b9cbde85494cdd8f84b [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a7d13d41ab9648f5b3b5884352620c8b" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 43801 } } peers { permanent_uuid: "915d2b26a77b4d28a0885933db7c65e7" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 36315 } } peers { permanent_uuid: "d9b9661d8c674b9cbde85494cdd8f84b" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 45215 } }
I20260620 14:50:33.717967 14078 leader_election.cc:288] T 00000000000000000000000000000000 P d9b9661d8c674b9cbde85494cdd8f84b [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers a7d13d41ab9648f5b3b5884352620c8b (127.0.97.62:43801), 915d2b26a77b4d28a0885933db7c65e7 (127.0.97.61:36315)
I20260620 14:50:33.719292 13966 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "d9b9661d8c674b9cbde85494cdd8f84b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "915d2b26a77b4d28a0885933db7c65e7" is_pre_election: true
I20260620 14:50:33.719944 13966 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 915d2b26a77b4d28a0885933db7c65e7 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d9b9661d8c674b9cbde85494cdd8f84b in term 0.
I20260620 14:50:33.721028 13892 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "d9b9661d8c674b9cbde85494cdd8f84b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a7d13d41ab9648f5b3b5884352620c8b" is_pre_election: true
I20260620 14:50:33.721638 13892 raft_consensus.cc:2454] T 00000000000000000000000000000000 P a7d13d41ab9648f5b3b5884352620c8b [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d9b9661d8c674b9cbde85494cdd8f84b in term 0.
I20260620 14:50:33.722307 14014 leader_election.cc:302] T 00000000000000000000000000000000 P d9b9661d8c674b9cbde85494cdd8f84b [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 915d2b26a77b4d28a0885933db7c65e7, d9b9661d8c674b9cbde85494cdd8f84b; no voters:
I20260620 14:50:33.723284 14078 raft_consensus.cc:2781] T 00000000000000000000000000000000 P d9b9661d8c674b9cbde85494cdd8f84b [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260620 14:50:33.723449 14078 raft_consensus.cc:482] T 00000000000000000000000000000000 P d9b9661d8c674b9cbde85494cdd8f84b [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260620 14:50:33.723551 14078 raft_consensus.cc:3037] T 00000000000000000000000000000000 P d9b9661d8c674b9cbde85494cdd8f84b [term 0 FOLLOWER]: Advancing to term 1
I20260620 14:50:33.726994 14078 raft_consensus.cc:504] T 00000000000000000000000000000000 P d9b9661d8c674b9cbde85494cdd8f84b [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a7d13d41ab9648f5b3b5884352620c8b" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 43801 } } peers { permanent_uuid: "915d2b26a77b4d28a0885933db7c65e7" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 36315 } } peers { permanent_uuid: "d9b9661d8c674b9cbde85494cdd8f84b" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 45215 } }
I20260620 14:50:33.728334 13892 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "d9b9661d8c674b9cbde85494cdd8f84b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a7d13d41ab9648f5b3b5884352620c8b"
I20260620 14:50:33.728574 13892 raft_consensus.cc:3037] T 00000000000000000000000000000000 P a7d13d41ab9648f5b3b5884352620c8b [term 0 FOLLOWER]: Advancing to term 1
I20260620 14:50:33.733531 14078 leader_election.cc:288] T 00000000000000000000000000000000 P d9b9661d8c674b9cbde85494cdd8f84b [CANDIDATE]: Term 1 election: Requested vote from peers a7d13d41ab9648f5b3b5884352620c8b (127.0.97.62:43801), 915d2b26a77b4d28a0885933db7c65e7 (127.0.97.61:36315)
I20260620 14:50:33.735005 13892 raft_consensus.cc:2454] T 00000000000000000000000000000000 P a7d13d41ab9648f5b3b5884352620c8b [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d9b9661d8c674b9cbde85494cdd8f84b in term 1.
I20260620 14:50:33.735109 13966 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "d9b9661d8c674b9cbde85494cdd8f84b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "915d2b26a77b4d28a0885933db7c65e7"
I20260620 14:50:33.735357 13966 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 915d2b26a77b4d28a0885933db7c65e7 [term 0 FOLLOWER]: Advancing to term 1
I20260620 14:50:33.738687 14017 leader_election.cc:302] T 00000000000000000000000000000000 P d9b9661d8c674b9cbde85494cdd8f84b [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: a7d13d41ab9648f5b3b5884352620c8b, d9b9661d8c674b9cbde85494cdd8f84b; no voters:
I20260620 14:50:33.741493 13966 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 915d2b26a77b4d28a0885933db7c65e7 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d9b9661d8c674b9cbde85494cdd8f84b in term 1.
I20260620 14:50:33.742333 14078 raft_consensus.cc:2781] T 00000000000000000000000000000000 P d9b9661d8c674b9cbde85494cdd8f84b [term 1 FOLLOWER]: Leader election won for term 1
I20260620 14:50:33.759902 14078 raft_consensus.cc:686] T 00000000000000000000000000000000 P d9b9661d8c674b9cbde85494cdd8f84b [term 1 LEADER]: Becoming Leader. State: Replica: d9b9661d8c674b9cbde85494cdd8f84b, State: Running, Role: LEADER
I20260620 14:50:33.760730 14078 consensus_queue.cc:237] T 00000000000000000000000000000000 P d9b9661d8c674b9cbde85494cdd8f84b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a7d13d41ab9648f5b3b5884352620c8b" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 43801 } } peers { permanent_uuid: "915d2b26a77b4d28a0885933db7c65e7" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 36315 } } peers { permanent_uuid: "d9b9661d8c674b9cbde85494cdd8f84b" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 45215 } }
I20260620 14:50:33.773564 14120 sys_catalog.cc:455] T 00000000000000000000000000000000 P d9b9661d8c674b9cbde85494cdd8f84b [sys.catalog]: SysCatalogTable state changed. Reason: New leader d9b9661d8c674b9cbde85494cdd8f84b. Latest consensus state: current_term: 1 leader_uuid: "d9b9661d8c674b9cbde85494cdd8f84b" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a7d13d41ab9648f5b3b5884352620c8b" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 43801 } } peers { permanent_uuid: "915d2b26a77b4d28a0885933db7c65e7" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 36315 } } peers { permanent_uuid: "d9b9661d8c674b9cbde85494cdd8f84b" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 45215 } } }
I20260620 14:50:33.773952 14120 sys_catalog.cc:458] T 00000000000000000000000000000000 P d9b9661d8c674b9cbde85494cdd8f84b [sys.catalog]: This master's current role is: LEADER
I20260620 14:50:33.775565 14122 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 14:50:33.777987 14122 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 14:50:33.807363 13966 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 915d2b26a77b4d28a0885933db7c65e7 [term 1 FOLLOWER]: Refusing update from remote peer d9b9661d8c674b9cbde85494cdd8f84b: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 14:50:33.808367 13892 raft_consensus.cc:1261] T 00000000000000000000000000000000 P a7d13d41ab9648f5b3b5884352620c8b [term 1 FOLLOWER]: Refusing update from remote peer d9b9661d8c674b9cbde85494cdd8f84b: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 14:50:33.809561 14120 consensus_queue.cc:1035] T 00000000000000000000000000000000 P d9b9661d8c674b9cbde85494cdd8f84b [LEADER]: Connected to new peer: Peer: permanent_uuid: "a7d13d41ab9648f5b3b5884352620c8b" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 43801 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 14:50:33.810161 14078 consensus_queue.cc:1035] T 00000000000000000000000000000000 P d9b9661d8c674b9cbde85494cdd8f84b [LEADER]: Connected to new peer: Peer: permanent_uuid: "915d2b26a77b4d28a0885933db7c65e7" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 36315 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 14:50:33.832779 14093 sys_catalog.cc:455] T 00000000000000000000000000000000 P a7d13d41ab9648f5b3b5884352620c8b [sys.catalog]: SysCatalogTable state changed. Reason: New leader d9b9661d8c674b9cbde85494cdd8f84b. Latest consensus state: current_term: 1 leader_uuid: "d9b9661d8c674b9cbde85494cdd8f84b" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a7d13d41ab9648f5b3b5884352620c8b" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 43801 } } peers { permanent_uuid: "915d2b26a77b4d28a0885933db7c65e7" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 36315 } } peers { permanent_uuid: "d9b9661d8c674b9cbde85494cdd8f84b" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 45215 } } }
I20260620 14:50:33.833123 14093 sys_catalog.cc:458] T 00000000000000000000000000000000 P a7d13d41ab9648f5b3b5884352620c8b [sys.catalog]: This master's current role is: FOLLOWER
I20260620 14:50:33.861138 14107 sys_catalog.cc:455] T 00000000000000000000000000000000 P 915d2b26a77b4d28a0885933db7c65e7 [sys.catalog]: SysCatalogTable state changed. Reason: New leader d9b9661d8c674b9cbde85494cdd8f84b. Latest consensus state: current_term: 1 leader_uuid: "d9b9661d8c674b9cbde85494cdd8f84b" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a7d13d41ab9648f5b3b5884352620c8b" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 43801 } } peers { permanent_uuid: "915d2b26a77b4d28a0885933db7c65e7" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 36315 } } peers { permanent_uuid: "d9b9661d8c674b9cbde85494cdd8f84b" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 45215 } } }
I20260620 14:50:33.861523 14107 sys_catalog.cc:458] T 00000000000000000000000000000000 P 915d2b26a77b4d28a0885933db7c65e7 [sys.catalog]: This master's current role is: FOLLOWER
I20260620 14:50:33.864370 14093 sys_catalog.cc:455] T 00000000000000000000000000000000 P a7d13d41ab9648f5b3b5884352620c8b [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "d9b9661d8c674b9cbde85494cdd8f84b" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a7d13d41ab9648f5b3b5884352620c8b" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 43801 } } peers { permanent_uuid: "915d2b26a77b4d28a0885933db7c65e7" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 36315 } } peers { permanent_uuid: "d9b9661d8c674b9cbde85494cdd8f84b" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 45215 } } }
I20260620 14:50:33.864697 14093 sys_catalog.cc:458] T 00000000000000000000000000000000 P a7d13d41ab9648f5b3b5884352620c8b [sys.catalog]: This master's current role is: FOLLOWER
I20260620 14:50:33.867091 14078 sys_catalog.cc:455] T 00000000000000000000000000000000 P d9b9661d8c674b9cbde85494cdd8f84b [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "d9b9661d8c674b9cbde85494cdd8f84b" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a7d13d41ab9648f5b3b5884352620c8b" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 43801 } } peers { permanent_uuid: "915d2b26a77b4d28a0885933db7c65e7" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 36315 } } peers { permanent_uuid: "d9b9661d8c674b9cbde85494cdd8f84b" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 45215 } } }
I20260620 14:50:33.867393 14078 sys_catalog.cc:458] T 00000000000000000000000000000000 P d9b9661d8c674b9cbde85494cdd8f84b [sys.catalog]: This master's current role is: LEADER
I20260620 14:50:33.868410 14122 catalog_manager.cc:1347] Generated new cluster ID: ed524e33c27f4fd7ae733a655e0b3108
I20260620 14:50:33.868508 14122 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 14:50:33.869630 14120 sys_catalog.cc:455] T 00000000000000000000000000000000 P d9b9661d8c674b9cbde85494cdd8f84b [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "d9b9661d8c674b9cbde85494cdd8f84b" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a7d13d41ab9648f5b3b5884352620c8b" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 43801 } } peers { permanent_uuid: "915d2b26a77b4d28a0885933db7c65e7" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 36315 } } peers { permanent_uuid: "d9b9661d8c674b9cbde85494cdd8f84b" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 45215 } } }
I20260620 14:50:33.869877 14120 sys_catalog.cc:458] T 00000000000000000000000000000000 P d9b9661d8c674b9cbde85494cdd8f84b [sys.catalog]: This master's current role is: LEADER
I20260620 14:50:33.879551 14125 mvcc.cc:205] Tried to move back new op lower bound from 7298936970460880896 to 7298936970290307072. Current Snapshot: MvccSnapshot[applied={T|T < 7298936970460880896}]
I20260620 14:50:33.892709 14107 sys_catalog.cc:455] T 00000000000000000000000000000000 P 915d2b26a77b4d28a0885933db7c65e7 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "d9b9661d8c674b9cbde85494cdd8f84b" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a7d13d41ab9648f5b3b5884352620c8b" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 43801 } } peers { permanent_uuid: "915d2b26a77b4d28a0885933db7c65e7" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 36315 } } peers { permanent_uuid: "d9b9661d8c674b9cbde85494cdd8f84b" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 45215 } } }
I20260620 14:50:33.892971 14107 sys_catalog.cc:458] T 00000000000000000000000000000000 P 915d2b26a77b4d28a0885933db7c65e7 [sys.catalog]: This master's current role is: FOLLOWER
I20260620 14:50:33.900656 14122 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 14:50:33.902619 14122 catalog_manager.cc:1504] Loading token signing keys...
I20260620 14:50:33.924489 14122 catalog_manager.cc:5868] T 00000000000000000000000000000000 P d9b9661d8c674b9cbde85494cdd8f84b: Generated new TSK 0
I20260620 14:50:33.925730 14122 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260620 14:50:34.602869 14104 catalog_manager.cc:1259] Loaded cluster ID: ed524e33c27f4fd7ae733a655e0b3108
I20260620 14:50:34.602988 14104 catalog_manager.cc:1552] T 00000000000000000000000000000000 P a7d13d41ab9648f5b3b5884352620c8b: loading cluster ID for follower catalog manager: success
I20260620 14:50:34.605512 14104 catalog_manager.cc:1574] T 00000000000000000000000000000000 P a7d13d41ab9648f5b3b5884352620c8b: acquiring CA information for follower catalog manager: success
I20260620 14:50:34.607405 14104 catalog_manager.cc:1602] T 00000000000000000000000000000000 P a7d13d41ab9648f5b3b5884352620c8b: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260620 14:50:34.718510 14118 catalog_manager.cc:1259] Loaded cluster ID: ed524e33c27f4fd7ae733a655e0b3108
I20260620 14:50:34.718676 14118 catalog_manager.cc:1552] T 00000000000000000000000000000000 P 915d2b26a77b4d28a0885933db7c65e7: loading cluster ID for follower catalog manager: success
I20260620 14:50:34.721362 14118 catalog_manager.cc:1574] T 00000000000000000000000000000000 P 915d2b26a77b4d28a0885933db7c65e7: acquiring CA information for follower catalog manager: success
I20260620 14:50:34.723281 14118 catalog_manager.cc:1602] T 00000000000000000000000000000000 P 915d2b26a77b4d28a0885933db7c65e7: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260620 14:50:35.072553 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskUci3DU/build/asan/bin/kudu pid 13850
I20260620 14:50:35.073900 13912 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 14:50:35.183158 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskUci3DU/build/asan/bin/kudu with pid 13850
I20260620 14:50:35.204495 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskUci3DU/build/asan/bin/kudu pid 13924
I20260620 14:50:35.205638 13986 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 14:50:35.312803 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskUci3DU/build/asan/bin/kudu with pid 13924
I20260620 14:50:35.333837 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskUci3DU/build/asan/bin/kudu pid 13998
I20260620 14:50:35.335131 14060 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 14:50:35.447305 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskUci3DU/build/asan/bin/kudu with pid 13998
2026-06-20T14:50:35Z chronyd exiting
[ OK ] SecurityITest.IPKICACert (4856 ms)
[ RUN ] SecurityITest.IPKICACertWebServerEndPoints
Loading random data
Initializing database '/tmp/dist-test-taskUci3DU/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1781966829091300-388-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 20 14:50:35 dist-test-slave-q9j6 krb5kdc[14141](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 20 14:50:35 dist-test-slave-q9j6 krb5kdc[14141](info): set up 2 sockets
Jun 20 14:50:35 dist-test-slave-q9j6 krb5kdc[14141](info): commencing operation
krb5kdc: starting...
W20260620 14:50:36.540290 388 mini_kdc.cc:121] Time spent starting KDC: real 1.054s user 0.008s sys 0.016s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskUci3DU/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1781966829091300-388-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskUci3DU/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1781966829091300-388-0/krb5kdc/test-user.keytab.
Jun 20 14:50:36 dist-test-slave-q9j6 krb5kdc[14141](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781967036, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-20T14:50:36Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T14:50:36Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.97.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskUci3DU/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1781966829091300-388-0/krb5kdc/kudu_127.0.97.62.keytab.
Entry for principal kudu/127.0.97.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskUci3DU/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1781966829091300-388-0/krb5kdc/kudu_127.0.97.62.keytab.
I20260620 14:50:36.804917 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskUci3DU/build/asan/bin/kudu
/tmp/dist-test-taskUci3DU/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskUci3DU/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1781966829091300-388-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskUci3DU/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1781966829091300-388-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskUci3DU/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1781966829091300-388-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskUci3DU/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1781966829091300-388-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:35209
--webserver_interface=127.0.97.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskUci3DU/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1781966829091300-388-0/krb5kdc/kudu_127.0.97.62.keytab
--principal=kudu/127.0.97.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:45143
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.97.62:35209
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskUci3DU/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1781966829091300-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskUci3DU/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1781966829091300-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskUci3DU/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1781966829091300-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 14:50:37.148237 14154 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 14:50:37.148610 14154 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 14:50:37.148778 14154 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 14:50:37.156416 14154 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 14:50:37.156546 14154 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 14:50:37.156674 14154 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 14:50:37.156750 14154 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 14:50:37.156843 14154 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 14:50:37.167246 14154 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:45143
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskUci3DU/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1781966829091300-388-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskUci3DU/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1781966829091300-388-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.97.62:35209
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskUci3DU/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1781966829091300-388-0/krb5kdc/kudu_127.0.97.62.keytab
--principal=kudu/127.0.97.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:35209
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskUci3DU/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1781966829091300-388-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.97.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskUci3DU/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1781966829091300-388-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 13:41:20 UTC on ccycloud.quasar-vdagoi.root.comops.site
build id 4457462
ASAN enabled
I20260620 14:50:37.168779 14154 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 14:50:37.170244 14154 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 14:50:37.182955 14159 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 14:50:37.183017 14162 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 14:50:37.183048 14160 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 14:50:37.183877 14154 server_base.cc:1034] running on GCE node
I20260620 14:50:37.184724 14154 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 14:50:37.186372 14154 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 14:50:37.187556 14154 hybrid_clock.cc:648] HybridClock initialized: now 1781967037187510 us; error 40 us; skew 500 ppm
I20260620 14:50:37.187966 14154 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 14:50:37 dist-test-slave-q9j6 krb5kdc[14141](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781967037, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 14:50:37.191272 14154 init.cc:377] Logged in from keytab as kudu/127.0.97.62@KRBTEST.COM (short username kudu)
I20260620 14:50:37.192970 14154 webserver.cc:460] Webserver started at http://127.0.97.62:46017/ using document root <none> and password file <none>
I20260620 14:50:37.193769 14154 fs_manager.cc:359] Metadata directory not provided
I20260620 14:50:37.193928 14154 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 14:50:37.194270 14154 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 14:50:37.197558 14154 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskUci3DU/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1781966829091300-388-0/minicluster-data/master-0/data/instance:
uuid: "c87738251d7742d098e8571f9717cc6b"
format_stamp: "Formatted at 2026-06-20 14:50:37 on dist-test-slave-q9j6"
server_key: "e6ba73ef2bb14a240fbd26d89d6b9013"
server_key_iv: "bffba4d81da490cbae8d66aaf97200c5"
server_key_version: "encryptionkey@0"
I20260620 14:50:37.198410 14154 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskUci3DU/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1781966829091300-388-0/minicluster-data/master-0/wal/instance:
uuid: "c87738251d7742d098e8571f9717cc6b"
format_stamp: "Formatted at 2026-06-20 14:50:37 on dist-test-slave-q9j6"
server_key: "e6ba73ef2bb14a240fbd26d89d6b9013"
server_key_iv: "bffba4d81da490cbae8d66aaf97200c5"
server_key_version: "encryptionkey@0"
I20260620 14:50:37.204243 14154 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.003s sys 0.005s
I20260620 14:50:37.208022 14169 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 14:50:37.209548 14154 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260620 14:50:37.209770 14154 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskUci3DU/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1781966829091300-388-0/minicluster-data/master-0/data,/tmp/dist-test-taskUci3DU/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1781966829091300-388-0/minicluster-data/master-0/wal
uuid: "c87738251d7742d098e8571f9717cc6b"
format_stamp: "Formatted at 2026-06-20 14:50:37 on dist-test-slave-q9j6"
server_key: "e6ba73ef2bb14a240fbd26d89d6b9013"
server_key_iv: "bffba4d81da490cbae8d66aaf97200c5"
server_key_version: "encryptionkey@0"
I20260620 14:50:37.210000 14154 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskUci3DU/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1781966829091300-388-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskUci3DU/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1781966829091300-388-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskUci3DU/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1781966829091300-388-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 14:50:37.228274 14154 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 14:50:37.233147 14154 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 14:50:37.233403 14154 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 14:50:37.251502 14154 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.62:35209
I20260620 14:50:37.251525 14220 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.62:35209 every 8 connection(s)
I20260620 14:50:37.253095 14154 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskUci3DU/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1781966829091300-388-0/minicluster-data/master-0/data/info.pb
I20260620 14:50:37.257229 14221 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 14:50:37.259186 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskUci3DU/build/asan/bin/kudu as pid 14154
I20260620 14:50:37.259413 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskUci3DU/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1781966829091300-388-0/minicluster-data/master-0/wal/instance
I20260620 14:50:37.260111 388 external_mini_cluster.cc:1428] Setting key cc9059c5019b600e25970cf2b741ba39
I20260620 14:50:37.270265 14221 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c87738251d7742d098e8571f9717cc6b: Bootstrap starting.
I20260620 14:50:37.274499 14221 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P c87738251d7742d098e8571f9717cc6b: Neither blocks nor log segments found. Creating new log.
I20260620 14:50:37.275916 14221 log.cc:824] T 00000000000000000000000000000000 P c87738251d7742d098e8571f9717cc6b: Log is configured to *not* fsync() on all Append() calls
Jun 20 14:50:37 dist-test-slave-q9j6 krb5kdc[14141](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781967036, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260620 14:50:37.279312 14221 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c87738251d7742d098e8571f9717cc6b: No bootstrap required, opened a new log
I20260620 14:50:37.285127 14224 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:50:37.262030 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38666 (local address 127.0.97.62:35209)
0620 14:50:37.262902 (+ 872us) server_negotiation.cc:206] Beginning negotiation
0620 14:50:37.262915 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0620 14:50:37.262963 (+ 48us) server_negotiation.cc:407] Connection header received
0620 14:50:37.264803 (+ 1840us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:50:37.264830 (+ 27us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:50:37.265271 (+ 441us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:50:37.265613 (+ 342us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:50:37.267711 (+ 2098us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:50:37.269062 (+ 1351us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:50:37.271118 (+ 2056us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:50:37.271940 (+ 822us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:50:37.272126 (+ 186us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:50:37.277291 (+ 5165us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 14:50:37.277299 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 14:50:37.277318 (+ 19us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 14:50:37.277362 (+ 44us) server_negotiation.cc:922] calling sasl_server_start()
0620 14:50:37.280601 (+ 3239us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:50:37.281381 (+ 780us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:50:37.281395 (+ 14us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:50:37.281402 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:50:37.281589 (+ 187us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:50:37.282010 (+ 421us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:50:37.282021 (+ 11us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:50:37.282029 (+ 8us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:50:37.282623 (+ 594us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 14:50:37.282971 (+ 348us) server_negotiation.cc:1035] Waiting for connection context
0620 14:50:37.283466 (+ 495us) server_negotiation.cc:299] Negotiation successful
0620 14:50:37.283958 (+ 492us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":637,"thread_start_us":407,"threads_started":1}
I20260620 14:50:37.288136 14221 raft_consensus.cc:348] T 00000000000000000000000000000000 P c87738251d7742d098e8571f9717cc6b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c87738251d7742d098e8571f9717cc6b" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 35209 } }
I20260620 14:50:37.288563 14221 raft_consensus.cc:374] T 00000000000000000000000000000000 P c87738251d7742d098e8571f9717cc6b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 14:50:37.288710 14221 raft_consensus.cc:729] T 00000000000000000000000000000000 P c87738251d7742d098e8571f9717cc6b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c87738251d7742d098e8571f9717cc6b, State: Initialized, Role: FOLLOWER
I20260620 14:50:37.289320 14221 consensus_queue.cc:260] T 00000000000000000000000000000000 P c87738251d7742d098e8571f9717cc6b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c87738251d7742d098e8571f9717cc6b" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 35209 } }
I20260620 14:50:37.289598 14221 raft_consensus.cc:388] T 00000000000000000000000000000000 P c87738251d7742d098e8571f9717cc6b [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 14:50:37.289707 388 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260620 14:50:37.289747 14221 raft_consensus.cc:482] T 00000000000000000000000000000000 P c87738251d7742d098e8571f9717cc6b [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 14:50:37.289912 14221 raft_consensus.cc:3037] T 00000000000000000000000000000000 P c87738251d7742d098e8571f9717cc6b [term 0 FOLLOWER]: Advancing to term 1
I20260620 14:50:37.293093 14221 raft_consensus.cc:504] T 00000000000000000000000000000000 P c87738251d7742d098e8571f9717cc6b [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c87738251d7742d098e8571f9717cc6b" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 35209 } }
I20260620 14:50:37.293707 14221 leader_election.cc:302] T 00000000000000000000000000000000 P c87738251d7742d098e8571f9717cc6b [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: c87738251d7742d098e8571f9717cc6b; no voters:
I20260620 14:50:37.294318 14221 leader_election.cc:288] T 00000000000000000000000000000000 P c87738251d7742d098e8571f9717cc6b [CANDIDATE]: Term 1 election: Requested vote from peers
I20260620 14:50:37.294492 14226 raft_consensus.cc:2781] T 00000000000000000000000000000000 P c87738251d7742d098e8571f9717cc6b [term 1 FOLLOWER]: Leader election won for term 1
I20260620 14:50:37.295186 14226 raft_consensus.cc:686] T 00000000000000000000000000000000 P c87738251d7742d098e8571f9717cc6b [term 1 LEADER]: Becoming Leader. State: Replica: c87738251d7742d098e8571f9717cc6b, State: Running, Role: LEADER
I20260620 14:50:37.295779 14226 consensus_queue.cc:237] T 00000000000000000000000000000000 P c87738251d7742d098e8571f9717cc6b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c87738251d7742d098e8571f9717cc6b" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 35209 } }
I20260620 14:50:37.296563 14221 sys_catalog.cc:564] T 00000000000000000000000000000000 P c87738251d7742d098e8571f9717cc6b [sys.catalog]: configured and running, proceeding with master startup.
I20260620 14:50:37.298966 14228 sys_catalog.cc:455] T 00000000000000000000000000000000 P c87738251d7742d098e8571f9717cc6b [sys.catalog]: SysCatalogTable state changed. Reason: New leader c87738251d7742d098e8571f9717cc6b. Latest consensus state: current_term: 1 leader_uuid: "c87738251d7742d098e8571f9717cc6b" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c87738251d7742d098e8571f9717cc6b" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 35209 } } }
I20260620 14:50:37.299383 14228 sys_catalog.cc:458] T 00000000000000000000000000000000 P c87738251d7742d098e8571f9717cc6b [sys.catalog]: This master's current role is: LEADER
I20260620 14:50:37.299111 14227 sys_catalog.cc:455] T 00000000000000000000000000000000 P c87738251d7742d098e8571f9717cc6b [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "c87738251d7742d098e8571f9717cc6b" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c87738251d7742d098e8571f9717cc6b" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 35209 } } }
I20260620 14:50:37.299522 14227 sys_catalog.cc:458] T 00000000000000000000000000000000 P c87738251d7742d098e8571f9717cc6b [sys.catalog]: This master's current role is: LEADER
I20260620 14:50:37.301133 14235 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 14:50:37.307888 14235 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 14:50:37.317677 14235 catalog_manager.cc:1347] Generated new cluster ID: 1431d3e08f34485b86d2aed20ae43ab6
I20260620 14:50:37.317866 14235 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 14:50:37.344413 14235 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 14:50:37.346482 14235 catalog_manager.cc:1504] Loading token signing keys...
I20260620 14:50:37.357703 14235 catalog_manager.cc:5868] T 00000000000000000000000000000000 P c87738251d7742d098e8571f9717cc6b: Generated new TSK 0
I20260620 14:50:37.358632 14235 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260620 14:50:37.402952 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskUci3DU/build/asan/bin/kudu pid 14154
I20260620 14:50:37.404213 14216 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 14:50:37.502705 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskUci3DU/build/asan/bin/kudu with pid 14154
2026-06-20T14:50:37Z chronyd exiting
[ OK ] SecurityITest.IPKICACertWebServerEndPoints (2054 ms)
[ RUN ] SecurityITest.TestEnableAndDisableEncryptedIPKICert
Loading random data
Initializing database '/tmp/dist-test-taskUci3DU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781966829091300-388-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 20 14:50:37 dist-test-slave-q9j6 krb5kdc[14249](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 20 14:50:37 dist-test-slave-q9j6 krb5kdc[14249](info): set up 2 sockets
Jun 20 14:50:37 dist-test-slave-q9j6 krb5kdc[14249](info): commencing operation
krb5kdc: starting...
W20260620 14:50:38.604202 388 mini_kdc.cc:121] Time spent starting KDC: real 1.064s user 0.002s sys 0.025s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskUci3DU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781966829091300-388-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskUci3DU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781966829091300-388-0/krb5kdc/test-user.keytab.
Jun 20 14:50:38 dist-test-slave-q9j6 krb5kdc[14249](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781967038, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-20T14:50:38Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T14:50:38Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.97.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskUci3DU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781966829091300-388-0/krb5kdc/kudu_127.0.97.62.keytab.
Entry for principal kudu/127.0.97.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskUci3DU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781966829091300-388-0/krb5kdc/kudu_127.0.97.62.keytab.
I20260620 14:50:38.846580 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskUci3DU/build/asan/bin/kudu
/tmp/dist-test-taskUci3DU/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskUci3DU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781966829091300-388-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskUci3DU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781966829091300-388-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskUci3DU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781966829091300-388-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskUci3DU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781966829091300-388-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:33193
--webserver_interface=127.0.97.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskUci3DU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781966829091300-388-0/krb5kdc/kudu_127.0.97.62.keytab
--principal=kudu/127.0.97.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:46747
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.97.62:33193
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskUci3DU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781966829091300-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskUci3DU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781966829091300-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskUci3DU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781966829091300-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 14:50:39.182618 14262 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 14:50:39.182937 14262 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 14:50:39.183032 14262 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 14:50:39.190706 14262 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 14:50:39.190822 14262 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 14:50:39.190869 14262 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 14:50:39.190920 14262 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 14:50:39.190966 14262 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 14:50:39.201102 14262 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:46747
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskUci3DU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781966829091300-388-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskUci3DU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781966829091300-388-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.97.62:33193
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskUci3DU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781966829091300-388-0/krb5kdc/kudu_127.0.97.62.keytab
--principal=kudu/127.0.97.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:33193
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskUci3DU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781966829091300-388-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.97.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskUci3DU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781966829091300-388-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 13:41:20 UTC on ccycloud.quasar-vdagoi.root.comops.site
build id 4457462
ASAN enabled
I20260620 14:50:39.202423 14262 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 14:50:39.203728 14262 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 14:50:39.215827 14268 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 14:50:39.215883 14270 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 14:50:39.215903 14267 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 14:50:39.216883 14262 server_base.cc:1034] running on GCE node
I20260620 14:50:39.217736 14262 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 14:50:39.219329 14262 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 14:50:39.220553 14262 hybrid_clock.cc:648] HybridClock initialized: now 1781967039220497 us; error 55 us; skew 500 ppm
I20260620 14:50:39.220890 14262 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 14:50:39 dist-test-slave-q9j6 krb5kdc[14249](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781967039, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 14:50:39.224382 14262 init.cc:377] Logged in from keytab as kudu/127.0.97.62@KRBTEST.COM (short username kudu)
I20260620 14:50:39.226114 14262 webserver.cc:460] Webserver started at http://127.0.97.62:45149/ using document root <none> and password file <none>
I20260620 14:50:39.226882 14262 fs_manager.cc:359] Metadata directory not provided
I20260620 14:50:39.227027 14262 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 14:50:39.227362 14262 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 14:50:39.230808 14262 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskUci3DU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781966829091300-388-0/minicluster-data/master-0/data/instance:
uuid: "6629e744611e4470843f038539aebff5"
format_stamp: "Formatted at 2026-06-20 14:50:39 on dist-test-slave-q9j6"
server_key: "92fec065209290b45c346159a1a06aed"
server_key_iv: "3768cf0e4973befb91ba089285d6b9e3"
server_key_version: "encryptionkey@0"
I20260620 14:50:39.231781 14262 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskUci3DU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781966829091300-388-0/minicluster-data/master-0/wal/instance:
uuid: "6629e744611e4470843f038539aebff5"
format_stamp: "Formatted at 2026-06-20 14:50:39 on dist-test-slave-q9j6"
server_key: "92fec065209290b45c346159a1a06aed"
server_key_iv: "3768cf0e4973befb91ba089285d6b9e3"
server_key_version: "encryptionkey@0"
I20260620 14:50:39.237438 14262 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.004s sys 0.000s
I20260620 14:50:39.241287 14277 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 14:50:39.242825 14262 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260620 14:50:39.243162 14262 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskUci3DU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781966829091300-388-0/minicluster-data/master-0/data,/tmp/dist-test-taskUci3DU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781966829091300-388-0/minicluster-data/master-0/wal
uuid: "6629e744611e4470843f038539aebff5"
format_stamp: "Formatted at 2026-06-20 14:50:39 on dist-test-slave-q9j6"
server_key: "92fec065209290b45c346159a1a06aed"
server_key_iv: "3768cf0e4973befb91ba089285d6b9e3"
server_key_version: "encryptionkey@0"
I20260620 14:50:39.243433 14262 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskUci3DU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781966829091300-388-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskUci3DU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781966829091300-388-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskUci3DU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781966829091300-388-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 14:50:39.261293 14262 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 14:50:39.265836 14262 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 14:50:39.266152 14262 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 14:50:39.285517 14262 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.62:33193
I20260620 14:50:39.285548 14328 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.62:33193 every 8 connection(s)
I20260620 14:50:39.286934 14262 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskUci3DU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781966829091300-388-0/minicluster-data/master-0/data/info.pb
I20260620 14:50:39.289616 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskUci3DU/build/asan/bin/kudu as pid 14262
I20260620 14:50:39.289841 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskUci3DU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781966829091300-388-0/minicluster-data/master-0/wal/instance
I20260620 14:50:39.290524 388 external_mini_cluster.cc:1428] Setting key b8d4ea4f0ab8ba9e761e4b738b8a40c7
I20260620 14:50:39.290999 14329 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 14:50:39.306522 14329 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 6629e744611e4470843f038539aebff5: Bootstrap starting.
Jun 20 14:50:39 dist-test-slave-q9j6 krb5kdc[14249](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781967038, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260620 14:50:39.309924 14329 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 6629e744611e4470843f038539aebff5: Neither blocks nor log segments found. Creating new log.
I20260620 14:50:39.311092 14329 log.cc:824] T 00000000000000000000000000000000 P 6629e744611e4470843f038539aebff5: Log is configured to *not* fsync() on all Append() calls
I20260620 14:50:39.314169 14329 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 6629e744611e4470843f038539aebff5: No bootstrap required, opened a new log
I20260620 14:50:39.315374 14332 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:50:39.292590 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49372 (local address 127.0.97.62:33193)
0620 14:50:39.293407 (+ 817us) server_negotiation.cc:206] Beginning negotiation
0620 14:50:39.293419 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0620 14:50:39.293461 (+ 42us) server_negotiation.cc:407] Connection header received
0620 14:50:39.295762 (+ 2301us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:50:39.295788 (+ 26us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:50:39.296237 (+ 449us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:50:39.296595 (+ 358us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:50:39.298601 (+ 2006us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:50:39.300062 (+ 1461us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:50:39.301993 (+ 1931us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:50:39.302760 (+ 767us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:50:39.302912 (+ 152us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:50:39.307819 (+ 4907us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 14:50:39.307828 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 14:50:39.307843 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 14:50:39.307886 (+ 43us) server_negotiation.cc:922] calling sasl_server_start()
0620 14:50:39.311092 (+ 3206us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:50:39.311939 (+ 847us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:50:39.311955 (+ 16us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:50:39.311962 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:50:39.312148 (+ 186us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:50:39.312581 (+ 433us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:50:39.312584 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:50:39.312586 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:50:39.313141 (+ 555us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 14:50:39.313446 (+ 305us) server_negotiation.cc:1035] Waiting for connection context
0620 14:50:39.313904 (+ 458us) server_negotiation.cc:299] Negotiation successful
0620 14:50:39.314393 (+ 489us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":613,"thread_start_us":373,"threads_started":1}
I20260620 14:50:39.322086 14329 raft_consensus.cc:348] T 00000000000000000000000000000000 P 6629e744611e4470843f038539aebff5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6629e744611e4470843f038539aebff5" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 33193 } }
I20260620 14:50:39.322510 14329 raft_consensus.cc:374] T 00000000000000000000000000000000 P 6629e744611e4470843f038539aebff5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 14:50:39.322621 14329 raft_consensus.cc:729] T 00000000000000000000000000000000 P 6629e744611e4470843f038539aebff5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6629e744611e4470843f038539aebff5, State: Initialized, Role: FOLLOWER
I20260620 14:50:39.323333 14329 consensus_queue.cc:260] T 00000000000000000000000000000000 P 6629e744611e4470843f038539aebff5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6629e744611e4470843f038539aebff5" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 33193 } }
I20260620 14:50:39.323637 14329 raft_consensus.cc:388] T 00000000000000000000000000000000 P 6629e744611e4470843f038539aebff5 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 14:50:39.323802 14329 raft_consensus.cc:482] T 00000000000000000000000000000000 P 6629e744611e4470843f038539aebff5 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 14:50:39.323993 14329 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 6629e744611e4470843f038539aebff5 [term 0 FOLLOWER]: Advancing to term 1
I20260620 14:50:39.326184 14329 raft_consensus.cc:504] T 00000000000000000000000000000000 P 6629e744611e4470843f038539aebff5 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6629e744611e4470843f038539aebff5" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 33193 } }
I20260620 14:50:39.326699 14329 leader_election.cc:302] T 00000000000000000000000000000000 P 6629e744611e4470843f038539aebff5 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 6629e744611e4470843f038539aebff5; no voters:
I20260620 14:50:39.327335 14329 leader_election.cc:288] T 00000000000000000000000000000000 P 6629e744611e4470843f038539aebff5 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260620 14:50:39.327467 14334 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 6629e744611e4470843f038539aebff5 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 14:50:39.328822 14334 raft_consensus.cc:686] T 00000000000000000000000000000000 P 6629e744611e4470843f038539aebff5 [term 1 LEADER]: Becoming Leader. State: Replica: 6629e744611e4470843f038539aebff5, State: Running, Role: LEADER
I20260620 14:50:39.329406 14334 consensus_queue.cc:237] T 00000000000000000000000000000000 P 6629e744611e4470843f038539aebff5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6629e744611e4470843f038539aebff5" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 33193 } }
I20260620 14:50:39.329838 14329 sys_catalog.cc:564] T 00000000000000000000000000000000 P 6629e744611e4470843f038539aebff5 [sys.catalog]: configured and running, proceeding with master startup.
I20260620 14:50:39.332722 14335 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6629e744611e4470843f038539aebff5 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "6629e744611e4470843f038539aebff5" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6629e744611e4470843f038539aebff5" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 33193 } } }
I20260620 14:50:39.332715 14336 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6629e744611e4470843f038539aebff5 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 6629e744611e4470843f038539aebff5. Latest consensus state: current_term: 1 leader_uuid: "6629e744611e4470843f038539aebff5" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6629e744611e4470843f038539aebff5" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 33193 } } }
I20260620 14:50:39.333137 14335 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6629e744611e4470843f038539aebff5 [sys.catalog]: This master's current role is: LEADER
I20260620 14:50:39.333215 14336 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6629e744611e4470843f038539aebff5 [sys.catalog]: This master's current role is: LEADER
I20260620 14:50:39.339591 14340 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 14:50:39.345180 14340 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 14:50:39.354123 14340 catalog_manager.cc:1347] Generated new cluster ID: 618885258ad7490b8093884920fa570e
I20260620 14:50:39.354250 14340 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 14:50:39.365828 14340 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 14:50:39.367980 14340 catalog_manager.cc:1504] Loading token signing keys...
I20260620 14:50:39.383127 14340 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 6629e744611e4470843f038539aebff5: Generated new TSK 0
I20260620 14:50:39.384385 14340 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.0.97.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskUci3DU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781966829091300-388-0/krb5kdc/kudu_127.0.97.1.keytab.
Entry for principal kudu/127.0.97.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskUci3DU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781966829091300-388-0/krb5kdc/kudu_127.0.97.1.keytab.
I20260620 14:50:39.447885 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskUci3DU/build/asan/bin/kudu
/tmp/dist-test-taskUci3DU/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskUci3DU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781966829091300-388-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskUci3DU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781966829091300-388-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskUci3DU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781966829091300-388-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskUci3DU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781966829091300-388-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.1:0
--local_ip_for_outbound_sockets=127.0.97.1
--webserver_interface=127.0.97.1
--webserver_port=0
--tserver_master_addrs=127.0.97.62:33193
--keytab_file=/tmp/dist-test-taskUci3DU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781966829091300-388-0/krb5kdc/kudu_127.0.97.1.keytab
--principal=kudu/127.0.97.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:46747
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskUci3DU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781966829091300-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskUci3DU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781966829091300-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskUci3DU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781966829091300-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 14:50:39.794385 14355 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 14:50:39.794792 14355 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 14:50:39.794966 14355 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 14:50:39.802459 14355 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 14:50:39.802621 14355 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 14:50:39.802802 14355 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.1
I20260620 14:50:39.813201 14355 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:46747
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskUci3DU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781966829091300-388-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskUci3DU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781966829091300-388-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskUci3DU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781966829091300-388-0/krb5kdc/kudu_127.0.97.1.keytab
--principal=kudu/127.0.97.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskUci3DU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781966829091300-388-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.97.1
--webserver_port=0
--tserver_master_addrs=127.0.97.62:33193
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.1
--log_dir=/tmp/dist-test-taskUci3DU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781966829091300-388-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 13:41:20 UTC on ccycloud.quasar-vdagoi.root.comops.site
build id 4457462
ASAN enabled
I20260620 14:50:39.814639 14355 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 14:50:39.816116 14355 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 14:50:39.830246 14361 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 14:50:39.830323 14363 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 14:50:39.830327 14360 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 14:50:39.831254 14355 server_base.cc:1034] running on GCE node
I20260620 14:50:39.831991 14355 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 14:50:39.833063 14355 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 14:50:39.834270 14355 hybrid_clock.cc:648] HybridClock initialized: now 1781967039834214 us; error 46 us; skew 500 ppm
I20260620 14:50:39.834617 14355 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 14:50:39 dist-test-slave-q9j6 krb5kdc[14249](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781967039, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 14:50:39.838263 14355 init.cc:377] Logged in from keytab as kudu/127.0.97.1@KRBTEST.COM (short username kudu)
I20260620 14:50:39.839960 14355 webserver.cc:460] Webserver started at http://127.0.97.1:34811/ using document root <none> and password file <none>
I20260620 14:50:39.840754 14355 fs_manager.cc:359] Metadata directory not provided
I20260620 14:50:39.840862 14355 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 14:50:39.841117 14355 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 14:50:39.844204 14355 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskUci3DU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781966829091300-388-0/minicluster-data/ts-0/data/instance:
uuid: "cadfcb134c47484cae04962cad62ee16"
format_stamp: "Formatted at 2026-06-20 14:50:39 on dist-test-slave-q9j6"
server_key: "58d469cb46839aee2d40e48feaf4b64d"
server_key_iv: "eb6548832933138fc21456ab31548ac5"
server_key_version: "encryptionkey@0"
I20260620 14:50:39.844957 14355 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskUci3DU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781966829091300-388-0/minicluster-data/ts-0/wal/instance:
uuid: "cadfcb134c47484cae04962cad62ee16"
format_stamp: "Formatted at 2026-06-20 14:50:39 on dist-test-slave-q9j6"
server_key: "58d469cb46839aee2d40e48feaf4b64d"
server_key_iv: "eb6548832933138fc21456ab31548ac5"
server_key_version: "encryptionkey@0"
I20260620 14:50:39.850608 14355 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.001s
I20260620 14:50:39.854130 14370 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 14:50:39.855713 14355 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260620 14:50:39.855911 14355 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskUci3DU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781966829091300-388-0/minicluster-data/ts-0/data,/tmp/dist-test-taskUci3DU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781966829091300-388-0/minicluster-data/ts-0/wal
uuid: "cadfcb134c47484cae04962cad62ee16"
format_stamp: "Formatted at 2026-06-20 14:50:39 on dist-test-slave-q9j6"
server_key: "58d469cb46839aee2d40e48feaf4b64d"
server_key_iv: "eb6548832933138fc21456ab31548ac5"
server_key_version: "encryptionkey@0"
I20260620 14:50:39.856158 14355 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskUci3DU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781966829091300-388-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskUci3DU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781966829091300-388-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskUci3DU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781966829091300-388-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 14:50:39.875516 14355 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 14:50:39.880239 14355 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 14:50:39.880534 14355 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 14:50:39.881532 14355 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 14:50:39.883044 14355 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 14:50:39.883118 14355 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 14:50:39.883272 14355 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 14:50:39.883347 14355 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 14:50:39.917096 14355 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.1:42731
I20260620 14:50:39.917140 14482 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.1:42731 every 8 connection(s)
I20260620 14:50:39.918479 14355 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskUci3DU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781966829091300-388-0/minicluster-data/ts-0/data/info.pb
I20260620 14:50:39.924252 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskUci3DU/build/asan/bin/kudu as pid 14355
I20260620 14:50:39.924469 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskUci3DU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781966829091300-388-0/minicluster-data/ts-0/wal/instance
I20260620 14:50:39.925323 388 external_mini_cluster.cc:1428] Setting key 72fe43e16ca9b0c4076acea5c0de9c67
Jun 20 14:50:39 dist-test-slave-q9j6 krb5kdc[14249](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781967039, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.1@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260620 14:50:39.942199 14485 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:50:39.921182 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.1:54891 (local address 127.0.97.62:33193)
0620 14:50:39.921741 (+ 559us) server_negotiation.cc:206] Beginning negotiation
0620 14:50:39.921748 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0620 14:50:39.922785 (+ 1037us) server_negotiation.cc:407] Connection header received
0620 14:50:39.924869 (+ 2084us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:50:39.924877 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:50:39.924993 (+ 116us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:50:39.925140 (+ 147us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:50:39.926706 (+ 1566us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:50:39.927726 (+ 1020us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:50:39.930396 (+ 2670us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:50:39.930905 (+ 509us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:50:39.931026 (+ 121us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:50:39.935538 (+ 4512us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 14:50:39.935551 (+ 13us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 14:50:39.935555 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 14:50:39.935593 (+ 38us) server_negotiation.cc:922] calling sasl_server_start()
0620 14:50:39.938363 (+ 2770us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:50:39.939471 (+ 1108us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:50:39.939475 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:50:39.939477 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:50:39.939675 (+ 198us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:50:39.940333 (+ 658us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:50:39.940336 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:50:39.940338 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:50:39.940613 (+ 275us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 14:50:39.940722 (+ 109us) server_negotiation.cc:1035] Waiting for connection context
0620 14:50:39.941736 (+ 1014us) server_negotiation.cc:299] Negotiation successful
0620 14:50:39.941925 (+ 189us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":416,"thread_start_us":279,"threads_started":1}
I20260620 14:50:39.943917 14486 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:50:39.921847 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:33193 (local address 127.0.97.1:54891)
0620 14:50:39.922598 (+ 751us) negotiation.cc:107] Waiting for socket to connect
0620 14:50:39.922639 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0620 14:50:39.924545 (+ 1906us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 14:50:39.925343 (+ 798us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 14:50:39.925352 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 14:50:39.925958 (+ 606us) client_negotiation.cc:189] Negotiated authn=SASL
0620 14:50:39.926532 (+ 574us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:50:39.926549 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:50:39.927913 (+ 1364us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:50:39.927918 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:50:39.930222 (+ 2304us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:50:39.930251 (+ 29us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:50:39.931180 (+ 929us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:50:39.931189 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:50:39.931551 (+ 362us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:50:39.932378 (+ 827us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 14:50:39.932422 (+ 44us) client_negotiation.cc:656] calling sasl_client_start()
0620 14:50:39.935244 (+ 2822us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 14:50:39.938553 (+ 3309us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 14:50:39.938566 (+ 13us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 14:50:39.938589 (+ 23us) client_negotiation.cc:760] calling sasl_client_step()
0620 14:50:39.939326 (+ 737us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 14:50:39.939820 (+ 494us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 14:50:39.939828 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 14:50:39.939834 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0620 14:50:39.940121 (+ 287us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 14:50:39.940764 (+ 643us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 14:50:39.940776 (+ 12us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 14:50:39.941303 (+ 527us) client_negotiation.cc:769] Sending connection context
0620 14:50:39.941652 (+ 349us) client_negotiation.cc:240] Negotiation successful
0620 14:50:39.942293 (+ 641us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":494,"thread_start_us":292,"threads_started":1}
I20260620 14:50:39.945685 14483 heartbeater.cc:347] Connected to a master server at 127.0.97.62:33193
I20260620 14:50:39.946024 14483 heartbeater.cc:464] Registering TS with master...
I20260620 14:50:39.946821 14483 heartbeater.cc:511] Master 127.0.97.62:33193 requested a full tablet report, sending...
I20260620 14:50:39.949756 14294 ts_manager.cc:194] Registered new tserver with Master: cadfcb134c47484cae04962cad62ee16 (127.0.97.1:42731)
WARNING: no policy specified for kudu/127.0.97.2@KRBTEST.COM; defaulting to no policy
I20260620 14:50:39.952076 14294 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.97.1@KRBTEST.COM'} at 127.0.97.1:54891
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskUci3DU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781966829091300-388-0/krb5kdc/kudu_127.0.97.2.keytab.
Entry for principal kudu/127.0.97.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskUci3DU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781966829091300-388-0/krb5kdc/kudu_127.0.97.2.keytab.
I20260620 14:50:39.980791 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskUci3DU/build/asan/bin/kudu
/tmp/dist-test-taskUci3DU/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskUci3DU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781966829091300-388-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskUci3DU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781966829091300-388-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskUci3DU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781966829091300-388-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskUci3DU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781966829091300-388-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.2:0
--local_ip_for_outbound_sockets=127.0.97.2
--webserver_interface=127.0.97.2
--webserver_port=0
--tserver_master_addrs=127.0.97.62:33193
--keytab_file=/tmp/dist-test-taskUci3DU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781966829091300-388-0/krb5kdc/kudu_127.0.97.2.keytab
--principal=kudu/127.0.97.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:46747
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskUci3DU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781966829091300-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskUci3DU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781966829091300-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskUci3DU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781966829091300-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 14:50:40.320637 14489 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 14:50:40.320947 14489 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 14:50:40.321112 14489 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 14:50:40.328823 14489 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 14:50:40.328959 14489 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 14:50:40.329141 14489 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.2
I20260620 14:50:40.339573 14489 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:46747
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskUci3DU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781966829091300-388-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskUci3DU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781966829091300-388-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskUci3DU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781966829091300-388-0/krb5kdc/kudu_127.0.97.2.keytab
--principal=kudu/127.0.97.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskUci3DU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781966829091300-388-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.97.2
--webserver_port=0
--tserver_master_addrs=127.0.97.62:33193
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.2
--log_dir=/tmp/dist-test-taskUci3DU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781966829091300-388-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 13:41:20 UTC on ccycloud.quasar-vdagoi.root.comops.site
build id 4457462
ASAN enabled
I20260620 14:50:40.341136 14489 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 14:50:40.342588 14489 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 14:50:40.356115 14495 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 14:50:40.356197 14494 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 14:50:40.356761 14497 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 14:50:40.357260 14489 server_base.cc:1034] running on GCE node
I20260620 14:50:40.357942 14489 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 14:50:40.358906 14489 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 14:50:40.360109 14489 hybrid_clock.cc:648] HybridClock initialized: now 1781967040360069 us; error 36 us; skew 500 ppm
I20260620 14:50:40.360549 14489 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 14:50:40 dist-test-slave-q9j6 krb5kdc[14249](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781967040, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 14:50:40.363948 14489 init.cc:377] Logged in from keytab as kudu/127.0.97.2@KRBTEST.COM (short username kudu)
I20260620 14:50:40.365373 14489 webserver.cc:460] Webserver started at http://127.0.97.2:37687/ using document root <none> and password file <none>
I20260620 14:50:40.366082 14489 fs_manager.cc:359] Metadata directory not provided
I20260620 14:50:40.366230 14489 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 14:50:40.366535 14489 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 14:50:40.369592 14489 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskUci3DU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781966829091300-388-0/minicluster-data/ts-1/data/instance:
uuid: "1dcd6c66dba34248933d56d9ba564a97"
format_stamp: "Formatted at 2026-06-20 14:50:40 on dist-test-slave-q9j6"
server_key: "41d9e2f20f0aecac40cb62e44dd073f3"
server_key_iv: "283ebcdfb4a4714b498de7161c900549"
server_key_version: "encryptionkey@0"
I20260620 14:50:40.370437 14489 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskUci3DU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781966829091300-388-0/minicluster-data/ts-1/wal/instance:
uuid: "1dcd6c66dba34248933d56d9ba564a97"
format_stamp: "Formatted at 2026-06-20 14:50:40 on dist-test-slave-q9j6"
server_key: "41d9e2f20f0aecac40cb62e44dd073f3"
server_key_iv: "283ebcdfb4a4714b498de7161c900549"
server_key_version: "encryptionkey@0"
I20260620 14:50:40.375844 14489 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.003s sys 0.004s
I20260620 14:50:40.379348 14504 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 14:50:40.380770 14489 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260620 14:50:40.380959 14489 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskUci3DU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781966829091300-388-0/minicluster-data/ts-1/data,/tmp/dist-test-taskUci3DU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781966829091300-388-0/minicluster-data/ts-1/wal
uuid: "1dcd6c66dba34248933d56d9ba564a97"
format_stamp: "Formatted at 2026-06-20 14:50:40 on dist-test-slave-q9j6"
server_key: "41d9e2f20f0aecac40cb62e44dd073f3"
server_key_iv: "283ebcdfb4a4714b498de7161c900549"
server_key_version: "encryptionkey@0"
I20260620 14:50:40.381171 14489 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskUci3DU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781966829091300-388-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskUci3DU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781966829091300-388-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskUci3DU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781966829091300-388-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 14:50:40.397287 14489 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 14:50:40.401986 14489 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 14:50:40.402333 14489 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 14:50:40.403443 14489 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 14:50:40.404886 14489 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 14:50:40.404991 14489 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 14:50:40.405156 14489 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 14:50:40.405229 14489 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 14:50:40.441812 14489 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.2:45359
I20260620 14:50:40.441872 14616 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.2:45359 every 8 connection(s)
I20260620 14:50:40.443199 14489 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskUci3DU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781966829091300-388-0/minicluster-data/ts-1/data/info.pb
I20260620 14:50:40.444186 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskUci3DU/build/asan/bin/kudu as pid 14489
I20260620 14:50:40.444370 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskUci3DU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781966829091300-388-0/minicluster-data/ts-1/wal/instance
I20260620 14:50:40.444984 388 external_mini_cluster.cc:1428] Setting key 6bf3c8d82520c6866ae148ce67fa59d9
Jun 20 14:50:40 dist-test-slave-q9j6 krb5kdc[14249](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781967040, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.2@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260620 14:50:40.468825 14619 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:50:40.446323 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.2:35175 (local address 127.0.97.62:33193)
0620 14:50:40.446822 (+ 499us) server_negotiation.cc:206] Beginning negotiation
0620 14:50:40.446826 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 14:50:40.448301 (+ 1475us) server_negotiation.cc:407] Connection header received
0620 14:50:40.451318 (+ 3017us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:50:40.451322 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:50:40.451398 (+ 76us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:50:40.451511 (+ 113us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:50:40.453995 (+ 2484us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:50:40.454734 (+ 739us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:50:40.457040 (+ 2306us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:50:40.457825 (+ 785us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:50:40.457982 (+ 157us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:50:40.462400 (+ 4418us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 14:50:40.462406 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 14:50:40.462409 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 14:50:40.462445 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0620 14:50:40.465193 (+ 2748us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:50:40.466241 (+ 1048us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:50:40.466245 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:50:40.466247 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:50:40.466377 (+ 130us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:50:40.466908 (+ 531us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:50:40.466911 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:50:40.466913 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:50:40.467240 (+ 327us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 14:50:40.467393 (+ 153us) server_negotiation.cc:1035] Waiting for connection context
0620 14:50:40.468357 (+ 964us) server_negotiation.cc:299] Negotiation successful
0620 14:50:40.468543 (+ 186us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":389,"thread_start_us":278,"threads_started":1}
I20260620 14:50:40.470515 14620 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:50:40.447057 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:33193 (local address 127.0.97.2:35175)
0620 14:50:40.448095 (+ 1038us) negotiation.cc:107] Waiting for socket to connect
0620 14:50:40.448161 (+ 66us) client_negotiation.cc:174] Beginning negotiation
0620 14:50:40.450965 (+ 2804us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 14:50:40.451920 (+ 955us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 14:50:40.451937 (+ 17us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 14:50:40.452817 (+ 880us) client_negotiation.cc:189] Negotiated authn=SASL
0620 14:50:40.453752 (+ 935us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:50:40.453783 (+ 31us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:50:40.454931 (+ 1148us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:50:40.454940 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:50:40.456851 (+ 1911us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:50:40.456865 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:50:40.458059 (+ 1194us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:50:40.458064 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:50:40.458299 (+ 235us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:50:40.459034 (+ 735us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 14:50:40.459057 (+ 23us) client_negotiation.cc:656] calling sasl_client_start()
0620 14:50:40.462111 (+ 3054us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 14:50:40.465356 (+ 3245us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 14:50:40.465369 (+ 13us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 14:50:40.465392 (+ 23us) client_negotiation.cc:760] calling sasl_client_step()
0620 14:50:40.466094 (+ 702us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 14:50:40.466499 (+ 405us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 14:50:40.466507 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 14:50:40.466513 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0620 14:50:40.466787 (+ 274us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 14:50:40.467395 (+ 608us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 14:50:40.467407 (+ 12us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 14:50:40.467930 (+ 523us) client_negotiation.cc:769] Sending connection context
0620 14:50:40.468282 (+ 352us) client_negotiation.cc:240] Negotiation successful
0620 14:50:40.468957 (+ 675us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":692,"thread_start_us":445,"threads_started":1}
WARNING: no policy specified for kudu/127.0.97.3@KRBTEST.COM; defaulting to no policy
I20260620 14:50:40.472211 14617 heartbeater.cc:347] Connected to a master server at 127.0.97.62:33193
I20260620 14:50:40.472528 14617 heartbeater.cc:464] Registering TS with master...
I20260620 14:50:40.473308 14617 heartbeater.cc:511] Master 127.0.97.62:33193 requested a full tablet report, sending...
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.3@KRBTEST.COM" created.
I20260620 14:50:40.474967 14294 ts_manager.cc:194] Registered new tserver with Master: 1dcd6c66dba34248933d56d9ba564a97 (127.0.97.2:45359)
I20260620 14:50:40.475862 14294 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.97.2@KRBTEST.COM'} at 127.0.97.2:35175
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskUci3DU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781966829091300-388-0/krb5kdc/kudu_127.0.97.3.keytab.
Entry for principal kudu/127.0.97.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskUci3DU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781966829091300-388-0/krb5kdc/kudu_127.0.97.3.keytab.
I20260620 14:50:40.502393 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskUci3DU/build/asan/bin/kudu
/tmp/dist-test-taskUci3DU/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskUci3DU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781966829091300-388-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskUci3DU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781966829091300-388-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskUci3DU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781966829091300-388-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskUci3DU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781966829091300-388-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.3:0
--local_ip_for_outbound_sockets=127.0.97.3
--webserver_interface=127.0.97.3
--webserver_port=0
--tserver_master_addrs=127.0.97.62:33193
--keytab_file=/tmp/dist-test-taskUci3DU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781966829091300-388-0/krb5kdc/kudu_127.0.97.3.keytab
--principal=kudu/127.0.97.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:46747
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskUci3DU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781966829091300-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskUci3DU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781966829091300-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskUci3DU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781966829091300-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 14:50:40.845000 14623 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 14:50:40.845306 14623 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 14:50:40.845490 14623 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 14:50:40.853086 14623 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 14:50:40.853245 14623 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 14:50:40.853434 14623 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.3
I20260620 14:50:40.864004 14623 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:46747
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskUci3DU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781966829091300-388-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskUci3DU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781966829091300-388-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskUci3DU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781966829091300-388-0/krb5kdc/kudu_127.0.97.3.keytab
--principal=kudu/127.0.97.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskUci3DU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781966829091300-388-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.97.3
--webserver_port=0
--tserver_master_addrs=127.0.97.62:33193
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.3
--log_dir=/tmp/dist-test-taskUci3DU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781966829091300-388-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 13:41:20 UTC on ccycloud.quasar-vdagoi.root.comops.site
build id 4457462
ASAN enabled
I20260620 14:50:40.865446 14623 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 14:50:40.866796 14623 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 14:50:40.880312 14628 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 14:50:40.880280 14631 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 14:50:40.880236 14629 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 14:50:40.881382 14623 server_base.cc:1034] running on GCE node
I20260620 14:50:40.881976 14623 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 14:50:40.882965 14623 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 14:50:40.884222 14623 hybrid_clock.cc:648] HybridClock initialized: now 1781967040884165 us; error 52 us; skew 500 ppm
I20260620 14:50:40.884562 14623 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 14:50:40 dist-test-slave-q9j6 krb5kdc[14249](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781967040, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 14:50:40.888171 14623 init.cc:377] Logged in from keytab as kudu/127.0.97.3@KRBTEST.COM (short username kudu)
I20260620 14:50:40.889735 14623 webserver.cc:460] Webserver started at http://127.0.97.3:33203/ using document root <none> and password file <none>
I20260620 14:50:40.890499 14623 fs_manager.cc:359] Metadata directory not provided
I20260620 14:50:40.890609 14623 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 14:50:40.890872 14623 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 14:50:40.893999 14623 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskUci3DU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781966829091300-388-0/minicluster-data/ts-2/data/instance:
uuid: "8fcc67d49a394238bc3c99b4560b78ec"
format_stamp: "Formatted at 2026-06-20 14:50:40 on dist-test-slave-q9j6"
server_key: "9787487b583dea31aca007219ece3219"
server_key_iv: "046b47dfdcc448e584d92f28422c36cd"
server_key_version: "encryptionkey@0"
I20260620 14:50:40.894804 14623 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskUci3DU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781966829091300-388-0/minicluster-data/ts-2/wal/instance:
uuid: "8fcc67d49a394238bc3c99b4560b78ec"
format_stamp: "Formatted at 2026-06-20 14:50:40 on dist-test-slave-q9j6"
server_key: "9787487b583dea31aca007219ece3219"
server_key_iv: "046b47dfdcc448e584d92f28422c36cd"
server_key_version: "encryptionkey@0"
I20260620 14:50:40.900346 14623 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.007s sys 0.000s
I20260620 14:50:40.904095 14638 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 14:50:40.905561 14623 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.001s
I20260620 14:50:40.905781 14623 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskUci3DU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781966829091300-388-0/minicluster-data/ts-2/data,/tmp/dist-test-taskUci3DU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781966829091300-388-0/minicluster-data/ts-2/wal
uuid: "8fcc67d49a394238bc3c99b4560b78ec"
format_stamp: "Formatted at 2026-06-20 14:50:40 on dist-test-slave-q9j6"
server_key: "9787487b583dea31aca007219ece3219"
server_key_iv: "046b47dfdcc448e584d92f28422c36cd"
server_key_version: "encryptionkey@0"
I20260620 14:50:40.906010 14623 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskUci3DU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781966829091300-388-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskUci3DU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781966829091300-388-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskUci3DU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781966829091300-388-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 14:50:40.930954 14623 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 14:50:40.935329 14623 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 14:50:40.935739 14623 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 14:50:40.936844 14623 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 14:50:40.938225 14623 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 14:50:40.938331 14623 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 14:50:40.938496 14623 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 14:50:40.938571 14623 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 14:50:40.955745 14483 heartbeater.cc:503] Master 127.0.97.62:33193 was elected leader, sending a full tablet report...
I20260620 14:50:40.972891 14623 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.3:35903
I20260620 14:50:40.972985 14750 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.3:35903 every 8 connection(s)
I20260620 14:50:40.974246 14623 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskUci3DU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781966829091300-388-0/minicluster-data/ts-2/data/info.pb
I20260620 14:50:40.975848 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskUci3DU/build/asan/bin/kudu as pid 14623
I20260620 14:50:40.976042 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskUci3DU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781966829091300-388-0/minicluster-data/ts-2/wal/instance
I20260620 14:50:40.976480 388 external_mini_cluster.cc:1428] Setting key bdad62517217c01b868a2d0bb4e41833
Jun 20 14:50:40 dist-test-slave-q9j6 krb5kdc[14249](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781967040, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.3@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260620 14:50:40.995414 14753 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:50:40.976981 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.3:53259 (local address 127.0.97.62:33193)
0620 14:50:40.977536 (+ 555us) server_negotiation.cc:206] Beginning negotiation
0620 14:50:40.977545 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0620 14:50:40.978660 (+ 1115us) server_negotiation.cc:407] Connection header received
0620 14:50:40.980959 (+ 2299us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:50:40.980964 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:50:40.981064 (+ 100us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:50:40.981201 (+ 137us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:50:40.982729 (+ 1528us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:50:40.983495 (+ 766us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:50:40.985210 (+ 1715us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:50:40.985726 (+ 516us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:50:40.985863 (+ 137us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:50:40.989441 (+ 3578us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 14:50:40.989450 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 14:50:40.989460 (+ 10us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 14:50:40.989496 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0620 14:50:40.992578 (+ 3082us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:50:40.993334 (+ 756us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:50:40.993338 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:50:40.993340 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:50:40.993462 (+ 122us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:50:40.993843 (+ 381us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:50:40.993845 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:50:40.993847 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:50:40.994117 (+ 270us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 14:50:40.994233 (+ 116us) server_negotiation.cc:1035] Waiting for connection context
0620 14:50:40.994906 (+ 673us) server_negotiation.cc:299] Negotiation successful
0620 14:50:40.995122 (+ 216us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":432,"thread_start_us":290,"threads_started":1}
I20260620 14:50:40.996407 14754 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:50:40.977631 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:33193 (local address 127.0.97.3:53259)
0620 14:50:40.978496 (+ 865us) negotiation.cc:107] Waiting for socket to connect
0620 14:50:40.978553 (+ 57us) client_negotiation.cc:174] Beginning negotiation
0620 14:50:40.980690 (+ 2137us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 14:50:40.981443 (+ 753us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 14:50:40.981452 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 14:50:40.981974 (+ 522us) client_negotiation.cc:189] Negotiated authn=SASL
0620 14:50:40.982552 (+ 578us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:50:40.982568 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:50:40.983691 (+ 1123us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:50:40.983695 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:50:40.985067 (+ 1372us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:50:40.985079 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:50:40.985951 (+ 872us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:50:40.985961 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:50:40.986263 (+ 302us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:50:40.986902 (+ 639us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 14:50:40.986924 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0620 14:50:40.989195 (+ 2271us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 14:50:40.992732 (+ 3537us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 14:50:40.992739 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 14:50:40.992755 (+ 16us) client_negotiation.cc:760] calling sasl_client_step()
0620 14:50:40.993203 (+ 448us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 14:50:40.993555 (+ 352us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 14:50:40.993558 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 14:50:40.993560 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 14:50:40.993743 (+ 183us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 14:50:40.994240 (+ 497us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 14:50:40.994246 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 14:50:40.994572 (+ 326us) client_negotiation.cc:769] Sending connection context
0620 14:50:40.994799 (+ 227us) client_negotiation.cc:240] Negotiation successful
0620 14:50:40.995242 (+ 443us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":587,"thread_start_us":353,"threads_started":1}
I20260620 14:50:40.997784 14751 heartbeater.cc:347] Connected to a master server at 127.0.97.62:33193
I20260620 14:50:40.998107 14751 heartbeater.cc:464] Registering TS with master...
I20260620 14:50:40.998835 14751 heartbeater.cc:511] Master 127.0.97.62:33193 requested a full tablet report, sending...
I20260620 14:50:41.000375 14294 ts_manager.cc:194] Registered new tserver with Master: 8fcc67d49a394238bc3c99b4560b78ec (127.0.97.3:35903)
I20260620 14:50:41.001210 14294 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.97.3@KRBTEST.COM'} at 127.0.97.3:53259
I20260620 14:50:41.005101 388 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260620 14:50:41.022636 14753 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:50:41.009790 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57762 (local address 127.0.97.62:33193)
0620 14:50:41.010038 (+ 248us) server_negotiation.cc:206] Beginning negotiation
0620 14:50:41.010045 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0620 14:50:41.010384 (+ 339us) server_negotiation.cc:407] Connection header received
0620 14:50:41.010561 (+ 177us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:50:41.010565 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:50:41.010642 (+ 77us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:50:41.010725 (+ 83us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:50:41.012281 (+ 1556us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:50:41.013019 (+ 738us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:50:41.014446 (+ 1427us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:50:41.014886 (+ 440us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:50:41.014988 (+ 102us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:50:41.017406 (+ 2418us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 14:50:41.017411 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 14:50:41.017415 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 14:50:41.017450 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0620 14:50:41.020142 (+ 2692us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:50:41.020830 (+ 688us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:50:41.020833 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:50:41.020836 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:50:41.020957 (+ 121us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:50:41.021264 (+ 307us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:50:41.021267 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:50:41.021269 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:50:41.021648 (+ 379us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 14:50:41.021887 (+ 239us) server_negotiation.cc:1035] Waiting for connection context
0620 14:50:41.022097 (+ 210us) server_negotiation.cc:299] Negotiation successful
0620 14:50:41.022287 (+ 190us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":76}
I20260620 14:50:41.026850 14294 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:57762:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260620 14:50:41.033962 14294 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260620 14:50:41.058773 14767 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:50:41.047216 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45090 (local address 127.0.97.3:35903)
0620 14:50:41.050007 (+ 2791us) server_negotiation.cc:206] Beginning negotiation
0620 14:50:41.050015 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0620 14:50:41.050044 (+ 29us) server_negotiation.cc:407] Connection header received
0620 14:50:41.050134 (+ 90us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:50:41.050143 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:50:41.050398 (+ 255us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:50:41.051277 (+ 879us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 14:50:41.051579 (+ 302us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:50:41.053303 (+ 1724us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:50:41.056322 (+ 3019us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:50:41.057402 (+ 1080us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:50:41.057575 (+ 173us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:50:41.057850 (+ 275us) server_negotiation.cc:1035] Waiting for connection context
0620 14:50:41.058203 (+ 353us) server_negotiation.cc:299] Negotiation successful
0620 14:50:41.058398 (+ 195us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":2632,"thread_start_us":2471,"threads_started":1}
I20260620 14:50:41.058867 14764 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:50:41.046721 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:42731 (local address 127.0.0.1:45890)
0620 14:50:41.047517 (+ 796us) negotiation.cc:107] Waiting for socket to connect
0620 14:50:41.047564 (+ 47us) client_negotiation.cc:174] Beginning negotiation
0620 14:50:41.047776 (+ 212us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 14:50:41.050390 (+ 2614us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 14:50:41.050398 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 14:50:41.050437 (+ 39us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 14:50:41.050966 (+ 529us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:50:41.050981 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:50:41.053455 (+ 2474us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:50:41.053461 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:50:41.055790 (+ 2329us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:50:41.055805 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:50:41.057807 (+ 2002us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:50:41.057814 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:50:41.058032 (+ 218us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:50:41.058067 (+ 35us) client_negotiation.cc:769] Sending connection context
0620 14:50:41.058139 (+ 72us) client_negotiation.cc:240] Negotiation successful
0620 14:50:41.058426 (+ 287us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":512,"mutex_wait_us":71,"thread_start_us":739,"threads_started":1}
I20260620 14:50:41.059203 14766 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:50:41.047225 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45890 (local address 127.0.97.1:42731)
0620 14:50:41.049667 (+ 2442us) server_negotiation.cc:206] Beginning negotiation
0620 14:50:41.049681 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0620 14:50:41.049714 (+ 33us) server_negotiation.cc:407] Connection header received
0620 14:50:41.049823 (+ 109us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:50:41.049834 (+ 11us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:50:41.050123 (+ 289us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:50:41.051307 (+ 1184us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 14:50:41.051579 (+ 272us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:50:41.053282 (+ 1703us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:50:41.055998 (+ 2716us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:50:41.057543 (+ 1545us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:50:41.058378 (+ 835us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:50:41.058619 (+ 241us) server_negotiation.cc:1035] Waiting for connection context
0620 14:50:41.058694 (+ 75us) server_negotiation.cc:299] Negotiation successful
0620 14:50:41.058865 (+ 171us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":2253,"thread_start_us":2090,"threads_started":1}
I20260620 14:50:41.059360 14768 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:50:41.049454 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.2:45359 (local address 127.0.0.1:49520)
0620 14:50:41.050398 (+ 944us) negotiation.cc:107] Waiting for socket to connect
0620 14:50:41.050419 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0620 14:50:41.050590 (+ 171us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 14:50:41.051766 (+ 1176us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 14:50:41.051771 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 14:50:41.051804 (+ 33us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 14:50:41.052257 (+ 453us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:50:41.052269 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:50:41.054091 (+ 1822us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:50:41.054098 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:50:41.056492 (+ 2394us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:50:41.056505 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:50:41.058417 (+ 1912us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:50:41.058423 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:50:41.058593 (+ 170us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:50:41.058625 (+ 32us) client_negotiation.cc:769] Sending connection context
0620 14:50:41.058693 (+ 68us) client_negotiation.cc:240] Negotiation successful
0620 14:50:41.058861 (+ 168us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":802,"mutex_wait_us":3,"thread_start_us":792,"threads_started":1}
I20260620 14:50:41.059908 14765 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:50:41.046879 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.3:35903 (local address 127.0.0.1:45090)
0620 14:50:41.048357 (+ 1478us) negotiation.cc:107] Waiting for socket to connect
0620 14:50:41.048375 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0620 14:50:41.048478 (+ 103us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 14:50:41.050610 (+ 2132us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 14:50:41.050621 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 14:50:41.050641 (+ 20us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 14:50:41.051082 (+ 441us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:50:41.051094 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:50:41.053494 (+ 2400us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:50:41.053497 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:50:41.056112 (+ 2615us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:50:41.056134 (+ 22us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:50:41.057617 (+ 1483us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:50:41.057623 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:50:41.057943 (+ 320us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:50:41.057989 (+ 46us) client_negotiation.cc:769] Sending connection context
0620 14:50:41.058134 (+ 145us) client_negotiation.cc:240] Negotiation successful
0620 14:50:41.059494 (+ 1360us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1354,"mutex_wait_us":179,"thread_start_us":1220,"threads_started":1}
I20260620 14:50:41.060943 14769 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:50:41.050037 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49520 (local address 127.0.97.2:45359)
0620 14:50:41.051121 (+ 1084us) server_negotiation.cc:206] Beginning negotiation
0620 14:50:41.051128 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0620 14:50:41.051178 (+ 50us) server_negotiation.cc:407] Connection header received
0620 14:50:41.051277 (+ 99us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:50:41.051284 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:50:41.051516 (+ 232us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:50:41.052467 (+ 951us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 14:50:41.052679 (+ 212us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:50:41.053914 (+ 1235us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:50:41.056661 (+ 2747us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:50:41.058190 (+ 1529us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:50:41.060002 (+ 1812us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:50:41.060270 (+ 268us) server_negotiation.cc:1035] Waiting for connection context
0620 14:50:41.060350 (+ 80us) server_negotiation.cc:299] Negotiation successful
0620 14:50:41.060582 (+ 232us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":940,"thread_start_us":804,"threads_started":1}
I20260620 14:50:41.063728 14686 tablet_service.cc:1467] Processing CreateTablet for tablet 9a096516e01d44b8944b6221b94df89e (DEFAULT_TABLE table=test-table [id=48b19ad7987447e4b1ff739c5eeccc2d]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 14:50:41.063766 14418 tablet_service.cc:1467] Processing CreateTablet for tablet 9a096516e01d44b8944b6221b94df89e (DEFAULT_TABLE table=test-table [id=48b19ad7987447e4b1ff739c5eeccc2d]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 14:50:41.064462 14552 tablet_service.cc:1467] Processing CreateTablet for tablet 9a096516e01d44b8944b6221b94df89e (DEFAULT_TABLE table=test-table [id=48b19ad7987447e4b1ff739c5eeccc2d]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 14:50:41.065119 14686 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9a096516e01d44b8944b6221b94df89e. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 14:50:41.065263 14418 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9a096516e01d44b8944b6221b94df89e. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 14:50:41.065717 14552 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9a096516e01d44b8944b6221b94df89e. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 14:50:41.077281 14770 tablet_bootstrap.cc:492] T 9a096516e01d44b8944b6221b94df89e P 8fcc67d49a394238bc3c99b4560b78ec: Bootstrap starting.
I20260620 14:50:41.080855 14771 tablet_bootstrap.cc:492] T 9a096516e01d44b8944b6221b94df89e P 1dcd6c66dba34248933d56d9ba564a97: Bootstrap starting.
I20260620 14:50:41.081707 14770 tablet_bootstrap.cc:654] T 9a096516e01d44b8944b6221b94df89e P 8fcc67d49a394238bc3c99b4560b78ec: Neither blocks nor log segments found. Creating new log.
I20260620 14:50:41.083173 14770 log.cc:824] T 9a096516e01d44b8944b6221b94df89e P 8fcc67d49a394238bc3c99b4560b78ec: Log is configured to *not* fsync() on all Append() calls
I20260620 14:50:41.084173 14772 tablet_bootstrap.cc:492] T 9a096516e01d44b8944b6221b94df89e P cadfcb134c47484cae04962cad62ee16: Bootstrap starting.
I20260620 14:50:41.084946 14771 tablet_bootstrap.cc:654] T 9a096516e01d44b8944b6221b94df89e P 1dcd6c66dba34248933d56d9ba564a97: Neither blocks nor log segments found. Creating new log.
I20260620 14:50:41.086381 14771 log.cc:824] T 9a096516e01d44b8944b6221b94df89e P 1dcd6c66dba34248933d56d9ba564a97: Log is configured to *not* fsync() on all Append() calls
I20260620 14:50:41.087953 14772 tablet_bootstrap.cc:654] T 9a096516e01d44b8944b6221b94df89e P cadfcb134c47484cae04962cad62ee16: Neither blocks nor log segments found. Creating new log.
I20260620 14:50:41.088016 14770 tablet_bootstrap.cc:492] T 9a096516e01d44b8944b6221b94df89e P 8fcc67d49a394238bc3c99b4560b78ec: No bootstrap required, opened a new log
I20260620 14:50:41.088397 14770 ts_tablet_manager.cc:1397] T 9a096516e01d44b8944b6221b94df89e P 8fcc67d49a394238bc3c99b4560b78ec: Time spent bootstrapping tablet: real 0.011s user 0.010s sys 0.000s
I20260620 14:50:41.089285 14772 log.cc:824] T 9a096516e01d44b8944b6221b94df89e P cadfcb134c47484cae04962cad62ee16: Log is configured to *not* fsync() on all Append() calls
I20260620 14:50:41.089697 14771 tablet_bootstrap.cc:492] T 9a096516e01d44b8944b6221b94df89e P 1dcd6c66dba34248933d56d9ba564a97: No bootstrap required, opened a new log
I20260620 14:50:41.090029 14771 ts_tablet_manager.cc:1397] T 9a096516e01d44b8944b6221b94df89e P 1dcd6c66dba34248933d56d9ba564a97: Time spent bootstrapping tablet: real 0.009s user 0.007s sys 0.000s
I20260620 14:50:41.092760 14772 tablet_bootstrap.cc:492] T 9a096516e01d44b8944b6221b94df89e P cadfcb134c47484cae04962cad62ee16: No bootstrap required, opened a new log
I20260620 14:50:41.093043 14772 ts_tablet_manager.cc:1397] T 9a096516e01d44b8944b6221b94df89e P cadfcb134c47484cae04962cad62ee16: Time spent bootstrapping tablet: real 0.009s user 0.004s sys 0.003s
I20260620 14:50:41.096488 14770 raft_consensus.cc:348] T 9a096516e01d44b8944b6221b94df89e P 8fcc67d49a394238bc3c99b4560b78ec [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cadfcb134c47484cae04962cad62ee16" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 42731 } } peers { permanent_uuid: "8fcc67d49a394238bc3c99b4560b78ec" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 35903 } } peers { permanent_uuid: "1dcd6c66dba34248933d56d9ba564a97" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 45359 } }
I20260620 14:50:41.096897 14770 raft_consensus.cc:374] T 9a096516e01d44b8944b6221b94df89e P 8fcc67d49a394238bc3c99b4560b78ec [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 14:50:41.097033 14770 raft_consensus.cc:729] T 9a096516e01d44b8944b6221b94df89e P 8fcc67d49a394238bc3c99b4560b78ec [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8fcc67d49a394238bc3c99b4560b78ec, State: Initialized, Role: FOLLOWER
I20260620 14:50:41.097690 14770 consensus_queue.cc:260] T 9a096516e01d44b8944b6221b94df89e P 8fcc67d49a394238bc3c99b4560b78ec [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cadfcb134c47484cae04962cad62ee16" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 42731 } } peers { permanent_uuid: "8fcc67d49a394238bc3c99b4560b78ec" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 35903 } } peers { permanent_uuid: "1dcd6c66dba34248933d56d9ba564a97" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 45359 } }
I20260620 14:50:41.099550 14770 ts_tablet_manager.cc:1428] T 9a096516e01d44b8944b6221b94df89e P 8fcc67d49a394238bc3c99b4560b78ec: Time spent starting tablet: real 0.011s user 0.010s sys 0.000s
I20260620 14:50:41.100322 14751 heartbeater.cc:503] Master 127.0.97.62:33193 was elected leader, sending a full tablet report...
I20260620 14:50:41.100620 14771 raft_consensus.cc:348] T 9a096516e01d44b8944b6221b94df89e P 1dcd6c66dba34248933d56d9ba564a97 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cadfcb134c47484cae04962cad62ee16" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 42731 } } peers { permanent_uuid: "8fcc67d49a394238bc3c99b4560b78ec" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 35903 } } peers { permanent_uuid: "1dcd6c66dba34248933d56d9ba564a97" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 45359 } }
I20260620 14:50:41.101055 14771 raft_consensus.cc:374] T 9a096516e01d44b8944b6221b94df89e P 1dcd6c66dba34248933d56d9ba564a97 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 14:50:41.101156 14771 raft_consensus.cc:729] T 9a096516e01d44b8944b6221b94df89e P 1dcd6c66dba34248933d56d9ba564a97 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1dcd6c66dba34248933d56d9ba564a97, State: Initialized, Role: FOLLOWER
I20260620 14:50:41.101797 14771 consensus_queue.cc:260] T 9a096516e01d44b8944b6221b94df89e P 1dcd6c66dba34248933d56d9ba564a97 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cadfcb134c47484cae04962cad62ee16" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 42731 } } peers { permanent_uuid: "8fcc67d49a394238bc3c99b4560b78ec" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 35903 } } peers { permanent_uuid: "1dcd6c66dba34248933d56d9ba564a97" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 45359 } }
I20260620 14:50:41.102985 14617 heartbeater.cc:503] Master 127.0.97.62:33193 was elected leader, sending a full tablet report...
I20260620 14:50:41.103711 14771 ts_tablet_manager.cc:1428] T 9a096516e01d44b8944b6221b94df89e P 1dcd6c66dba34248933d56d9ba564a97: Time spent starting tablet: real 0.014s user 0.013s sys 0.000s
I20260620 14:50:41.103786 14772 raft_consensus.cc:348] T 9a096516e01d44b8944b6221b94df89e P cadfcb134c47484cae04962cad62ee16 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cadfcb134c47484cae04962cad62ee16" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 42731 } } peers { permanent_uuid: "8fcc67d49a394238bc3c99b4560b78ec" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 35903 } } peers { permanent_uuid: "1dcd6c66dba34248933d56d9ba564a97" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 45359 } }
I20260620 14:50:41.104300 14772 raft_consensus.cc:374] T 9a096516e01d44b8944b6221b94df89e P cadfcb134c47484cae04962cad62ee16 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 14:50:41.104429 14772 raft_consensus.cc:729] T 9a096516e01d44b8944b6221b94df89e P cadfcb134c47484cae04962cad62ee16 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: cadfcb134c47484cae04962cad62ee16, State: Initialized, Role: FOLLOWER
I20260620 14:50:41.105108 14772 consensus_queue.cc:260] T 9a096516e01d44b8944b6221b94df89e P cadfcb134c47484cae04962cad62ee16 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cadfcb134c47484cae04962cad62ee16" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 42731 } } peers { permanent_uuid: "8fcc67d49a394238bc3c99b4560b78ec" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 35903 } } peers { permanent_uuid: "1dcd6c66dba34248933d56d9ba564a97" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 45359 } }
I20260620 14:50:41.107061 14772 ts_tablet_manager.cc:1428] T 9a096516e01d44b8944b6221b94df89e P cadfcb134c47484cae04962cad62ee16: Time spent starting tablet: real 0.014s user 0.010s sys 0.004s
W20260620 14:50:41.171088 14484 tablet.cc:2369] T 9a096516e01d44b8944b6221b94df89e P cadfcb134c47484cae04962cad62ee16: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260620 14:50:41.195451 14618 tablet.cc:2369] T 9a096516e01d44b8944b6221b94df89e P 1dcd6c66dba34248933d56d9ba564a97: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260620 14:50:41.226018 14752 tablet.cc:2369] T 9a096516e01d44b8944b6221b94df89e P 8fcc67d49a394238bc3c99b4560b78ec: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260620 14:50:41.353534 14777 raft_consensus.cc:482] T 9a096516e01d44b8944b6221b94df89e P 1dcd6c66dba34248933d56d9ba564a97 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260620 14:50:41.353813 14777 raft_consensus.cc:504] T 9a096516e01d44b8944b6221b94df89e P 1dcd6c66dba34248933d56d9ba564a97 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cadfcb134c47484cae04962cad62ee16" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 42731 } } peers { permanent_uuid: "8fcc67d49a394238bc3c99b4560b78ec" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 35903 } } peers { permanent_uuid: "1dcd6c66dba34248933d56d9ba564a97" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 45359 } }
I20260620 14:50:41.355629 14777 leader_election.cc:288] T 9a096516e01d44b8944b6221b94df89e P 1dcd6c66dba34248933d56d9ba564a97 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers cadfcb134c47484cae04962cad62ee16 (127.0.97.1:42731), 8fcc67d49a394238bc3c99b4560b78ec (127.0.97.3:35903)
I20260620 14:50:41.364877 14767 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:50:41.356242 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.2:39253 (local address 127.0.97.3:35903)
0620 14:50:41.356469 (+ 227us) server_negotiation.cc:206] Beginning negotiation
0620 14:50:41.356478 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0620 14:50:41.357157 (+ 679us) server_negotiation.cc:407] Connection header received
0620 14:50:41.357295 (+ 138us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:50:41.357300 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:50:41.357374 (+ 74us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:50:41.357458 (+ 84us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 14:50:41.358175 (+ 717us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:50:41.359284 (+ 1109us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:50:41.362336 (+ 3052us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:50:41.363648 (+ 1312us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:50:41.363801 (+ 153us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:50:41.363918 (+ 117us) server_negotiation.cc:1035] Waiting for connection context
0620 14:50:41.364453 (+ 535us) server_negotiation.cc:299] Negotiation successful
0620 14:50:41.364592 (+ 139us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":46}
I20260620 14:50:41.364877 14766 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:50:41.356102 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.2:49767 (local address 127.0.97.1:42731)
0620 14:50:41.356529 (+ 427us) server_negotiation.cc:206] Beginning negotiation
0620 14:50:41.356538 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0620 14:50:41.356878 (+ 340us) server_negotiation.cc:407] Connection header received
0620 14:50:41.357113 (+ 235us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:50:41.357119 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:50:41.357204 (+ 85us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:50:41.357301 (+ 97us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 14:50:41.357999 (+ 698us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:50:41.359169 (+ 1170us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:50:41.362333 (+ 3164us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:50:41.363656 (+ 1323us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:50:41.363801 (+ 145us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:50:41.363918 (+ 117us) server_negotiation.cc:1035] Waiting for connection context
0620 14:50:41.364453 (+ 535us) server_negotiation.cc:299] Negotiation successful
0620 14:50:41.364592 (+ 139us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":236}
I20260620 14:50:41.365113 14779 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:50:41.355826 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:42731 (local address 127.0.97.2:49767)
0620 14:50:41.356748 (+ 922us) negotiation.cc:107] Waiting for socket to connect
0620 14:50:41.356769 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0620 14:50:41.356965 (+ 196us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 14:50:41.357340 (+ 375us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 14:50:41.357344 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 14:50:41.357363 (+ 19us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 14:50:41.357839 (+ 476us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:50:41.357855 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:50:41.359345 (+ 1490us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:50:41.359354 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:50:41.362150 (+ 2796us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:50:41.362160 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:50:41.363960 (+ 1800us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:50:41.363967 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:50:41.364225 (+ 258us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:50:41.364305 (+ 80us) client_negotiation.cc:769] Sending connection context
0620 14:50:41.364404 (+ 99us) client_negotiation.cc:240] Negotiation successful
0620 14:50:41.364582 (+ 178us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":778,"mutex_wait_us":275,"spinlock_wait_cycles":41216,"thread_start_us":669,"threads_started":1}
I20260620 14:50:41.365135 14780 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:50:41.356109 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.3:35903 (local address 127.0.97.2:39253)
0620 14:50:41.357044 (+ 935us) negotiation.cc:107] Waiting for socket to connect
0620 14:50:41.357058 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0620 14:50:41.357165 (+ 107us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 14:50:41.357496 (+ 331us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 14:50:41.357507 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 14:50:41.357541 (+ 34us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 14:50:41.358001 (+ 460us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:50:41.358014 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:50:41.359409 (+ 1395us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:50:41.359413 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:50:41.362126 (+ 2713us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:50:41.362152 (+ 26us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:50:41.363962 (+ 1810us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:50:41.363967 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:50:41.364233 (+ 266us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:50:41.364297 (+ 64us) client_negotiation.cc:769] Sending connection context
0620 14:50:41.364405 (+ 108us) client_negotiation.cc:240] Negotiation successful
0620 14:50:41.364582 (+ 177us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":812,"mutex_wait_us":185,"thread_start_us":602,"threads_started":1}
I20260620 14:50:41.366526 14706 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9a096516e01d44b8944b6221b94df89e" candidate_uuid: "1dcd6c66dba34248933d56d9ba564a97" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8fcc67d49a394238bc3c99b4560b78ec" is_pre_election: true
I20260620 14:50:41.366523 14438 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9a096516e01d44b8944b6221b94df89e" candidate_uuid: "1dcd6c66dba34248933d56d9ba564a97" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "cadfcb134c47484cae04962cad62ee16" is_pre_election: true
I20260620 14:50:41.367148 14438 raft_consensus.cc:2454] T 9a096516e01d44b8944b6221b94df89e P cadfcb134c47484cae04962cad62ee16 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1dcd6c66dba34248933d56d9ba564a97 in term 0.
I20260620 14:50:41.367148 14706 raft_consensus.cc:2454] T 9a096516e01d44b8944b6221b94df89e P 8fcc67d49a394238bc3c99b4560b78ec [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1dcd6c66dba34248933d56d9ba564a97 in term 0.
I20260620 14:50:41.367975 14507 leader_election.cc:302] T 9a096516e01d44b8944b6221b94df89e P 1dcd6c66dba34248933d56d9ba564a97 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 3 yes votes; 0 no votes. yes voters: 1dcd6c66dba34248933d56d9ba564a97, 8fcc67d49a394238bc3c99b4560b78ec, cadfcb134c47484cae04962cad62ee16; no voters:
I20260620 14:50:41.368481 14777 raft_consensus.cc:2781] T 9a096516e01d44b8944b6221b94df89e P 1dcd6c66dba34248933d56d9ba564a97 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260620 14:50:41.368644 14777 raft_consensus.cc:482] T 9a096516e01d44b8944b6221b94df89e P 1dcd6c66dba34248933d56d9ba564a97 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260620 14:50:41.368752 14777 raft_consensus.cc:3037] T 9a096516e01d44b8944b6221b94df89e P 1dcd6c66dba34248933d56d9ba564a97 [term 0 FOLLOWER]: Advancing to term 1
I20260620 14:50:41.371014 14777 raft_consensus.cc:504] T 9a096516e01d44b8944b6221b94df89e P 1dcd6c66dba34248933d56d9ba564a97 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cadfcb134c47484cae04962cad62ee16" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 42731 } } peers { permanent_uuid: "8fcc67d49a394238bc3c99b4560b78ec" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 35903 } } peers { permanent_uuid: "1dcd6c66dba34248933d56d9ba564a97" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 45359 } }
I20260620 14:50:41.371827 14777 leader_election.cc:288] T 9a096516e01d44b8944b6221b94df89e P 1dcd6c66dba34248933d56d9ba564a97 [CANDIDATE]: Term 1 election: Requested vote from peers cadfcb134c47484cae04962cad62ee16 (127.0.97.1:42731), 8fcc67d49a394238bc3c99b4560b78ec (127.0.97.3:35903)
I20260620 14:50:41.372434 14438 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9a096516e01d44b8944b6221b94df89e" candidate_uuid: "1dcd6c66dba34248933d56d9ba564a97" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "cadfcb134c47484cae04962cad62ee16"
I20260620 14:50:41.372440 14706 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9a096516e01d44b8944b6221b94df89e" candidate_uuid: "1dcd6c66dba34248933d56d9ba564a97" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8fcc67d49a394238bc3c99b4560b78ec"
I20260620 14:50:41.372676 14706 raft_consensus.cc:3037] T 9a096516e01d44b8944b6221b94df89e P 8fcc67d49a394238bc3c99b4560b78ec [term 0 FOLLOWER]: Advancing to term 1
I20260620 14:50:41.372676 14438 raft_consensus.cc:3037] T 9a096516e01d44b8944b6221b94df89e P cadfcb134c47484cae04962cad62ee16 [term 0 FOLLOWER]: Advancing to term 1
I20260620 14:50:41.376057 14706 raft_consensus.cc:2454] T 9a096516e01d44b8944b6221b94df89e P 8fcc67d49a394238bc3c99b4560b78ec [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1dcd6c66dba34248933d56d9ba564a97 in term 1.
I20260620 14:50:41.376057 14438 raft_consensus.cc:2454] T 9a096516e01d44b8944b6221b94df89e P cadfcb134c47484cae04962cad62ee16 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1dcd6c66dba34248933d56d9ba564a97 in term 1.
I20260620 14:50:41.376619 14507 leader_election.cc:302] T 9a096516e01d44b8944b6221b94df89e P 1dcd6c66dba34248933d56d9ba564a97 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1dcd6c66dba34248933d56d9ba564a97, 8fcc67d49a394238bc3c99b4560b78ec; no voters:
I20260620 14:50:41.376976 14777 raft_consensus.cc:2781] T 9a096516e01d44b8944b6221b94df89e P 1dcd6c66dba34248933d56d9ba564a97 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 14:50:41.377493 14777 raft_consensus.cc:686] T 9a096516e01d44b8944b6221b94df89e P 1dcd6c66dba34248933d56d9ba564a97 [term 1 LEADER]: Becoming Leader. State: Replica: 1dcd6c66dba34248933d56d9ba564a97, State: Running, Role: LEADER
I20260620 14:50:41.378019 14777 consensus_queue.cc:237] T 9a096516e01d44b8944b6221b94df89e P 1dcd6c66dba34248933d56d9ba564a97 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cadfcb134c47484cae04962cad62ee16" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 42731 } } peers { permanent_uuid: "8fcc67d49a394238bc3c99b4560b78ec" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 35903 } } peers { permanent_uuid: "1dcd6c66dba34248933d56d9ba564a97" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 45359 } }
I20260620 14:50:41.382930 14293 catalog_manager.cc:5495] T 9a096516e01d44b8944b6221b94df89e P 1dcd6c66dba34248933d56d9ba564a97 reported cstate change: term changed from 0 to 1, leader changed from <none> to 1dcd6c66dba34248933d56d9ba564a97 (127.0.97.2). New cstate: current_term: 1 leader_uuid: "1dcd6c66dba34248933d56d9ba564a97" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cadfcb134c47484cae04962cad62ee16" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 42731 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "8fcc67d49a394238bc3c99b4560b78ec" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 35903 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "1dcd6c66dba34248933d56d9ba564a97" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 45359 } health_report { overall_health: HEALTHY } } }
I20260620 14:50:41.428647 14769 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:50:41.421961 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49528 (local address 127.0.97.2:45359)
0620 14:50:41.422202 (+ 241us) server_negotiation.cc:206] Beginning negotiation
0620 14:50:41.422209 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0620 14:50:41.422244 (+ 35us) server_negotiation.cc:407] Connection header received
0620 14:50:41.422353 (+ 109us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:50:41.422357 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:50:41.422449 (+ 92us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:50:41.422624 (+ 175us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 14:50:41.423277 (+ 653us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:50:41.424500 (+ 1223us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:50:41.426086 (+ 1586us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:50:41.426517 (+ 431us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:50:41.426669 (+ 152us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:50:41.427338 (+ 669us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 14:50:41.427990 (+ 652us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0620 14:50:41.428154 (+ 164us) server_negotiation.cc:1035] Waiting for connection context
0620 14:50:41.428243 (+ 89us) server_negotiation.cc:299] Negotiation successful
0620 14:50:41.428321 (+ 78us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":94}
I20260620 14:50:41.438499 14438 raft_consensus.cc:1261] T 9a096516e01d44b8944b6221b94df89e P cadfcb134c47484cae04962cad62ee16 [term 1 FOLLOWER]: Refusing update from remote peer 1dcd6c66dba34248933d56d9ba564a97: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 14:50:41.438504 14706 raft_consensus.cc:1261] T 9a096516e01d44b8944b6221b94df89e P 8fcc67d49a394238bc3c99b4560b78ec [term 1 FOLLOWER]: Refusing update from remote peer 1dcd6c66dba34248933d56d9ba564a97: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 14:50:41.439682 14781 consensus_queue.cc:1035] T 9a096516e01d44b8944b6221b94df89e P 1dcd6c66dba34248933d56d9ba564a97 [LEADER]: Connected to new peer: Peer: permanent_uuid: "cadfcb134c47484cae04962cad62ee16" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 42731 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 14:50:41.440097 14777 consensus_queue.cc:1035] T 9a096516e01d44b8944b6221b94df89e P 1dcd6c66dba34248933d56d9ba564a97 [LEADER]: Connected to new peer: Peer: permanent_uuid: "8fcc67d49a394238bc3c99b4560b78ec" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 35903 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 14:50:41.455171 14785 mvcc.cc:205] Tried to move back new op lower bound from 7298937001719816192 to 7298937001488134144. Current Snapshot: MvccSnapshot[applied={T|T < 7298937001719816192}]
I20260620 14:50:41.471698 14294 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:57762:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 14:50:41.472003 14294 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:57762:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 14:50:41.475835 14294 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 6629e744611e4470843f038539aebff5: Sending DeleteTablet for 3 replicas of tablet 9a096516e01d44b8944b6221b94df89e
I20260620 14:50:41.477191 14418 tablet_service.cc:1514] Processing DeleteTablet for tablet 9a096516e01d44b8944b6221b94df89e with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 14:50:41 UTC) from {username='kudu', principal='kudu/127.0.97.62@KRBTEST.COM'} at 127.0.0.1:45890
I20260620 14:50:41.477245 14686 tablet_service.cc:1514] Processing DeleteTablet for tablet 9a096516e01d44b8944b6221b94df89e with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 14:50:41 UTC) from {username='kudu', principal='kudu/127.0.97.62@KRBTEST.COM'} at 127.0.0.1:45090
I20260620 14:50:41.477680 14552 tablet_service.cc:1514] Processing DeleteTablet for tablet 9a096516e01d44b8944b6221b94df89e with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 14:50:41 UTC) from {username='kudu', principal='kudu/127.0.97.62@KRBTEST.COM'} at 127.0.0.1:49520
I20260620 14:50:41.478133 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskUci3DU/build/asan/bin/kudu pid 14355
I20260620 14:50:41.490731 14793 tablet_replica.cc:330] T 9a096516e01d44b8944b6221b94df89e P cadfcb134c47484cae04962cad62ee16: stopping tablet replica
I20260620 14:50:41.491350 14793 raft_consensus.cc:2229] T 9a096516e01d44b8944b6221b94df89e P cadfcb134c47484cae04962cad62ee16 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 14:50:41.492000 14793 raft_consensus.cc:2258] T 9a096516e01d44b8944b6221b94df89e P cadfcb134c47484cae04962cad62ee16 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 14:50:41.494683 14793 ts_tablet_manager.cc:1905] T 9a096516e01d44b8944b6221b94df89e P cadfcb134c47484cae04962cad62ee16: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 14:50:41.504138 14794 tablet_replica.cc:330] T 9a096516e01d44b8944b6221b94df89e P 8fcc67d49a394238bc3c99b4560b78ec: stopping tablet replica
I20260620 14:50:41.504683 14794 raft_consensus.cc:2229] T 9a096516e01d44b8944b6221b94df89e P 8fcc67d49a394238bc3c99b4560b78ec [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 14:50:41.505270 14794 raft_consensus.cc:2258] T 9a096516e01d44b8944b6221b94df89e P 8fcc67d49a394238bc3c99b4560b78ec [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 14:50:41.503589 14793 ts_tablet_manager.cc:1918] T 9a096516e01d44b8944b6221b94df89e P cadfcb134c47484cae04962cad62ee16: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260620 14:50:41.505743 14793 log.cc:1196] T 9a096516e01d44b8944b6221b94df89e P cadfcb134c47484cae04962cad62ee16: Deleting WAL directory at /tmp/dist-test-taskUci3DU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781966829091300-388-0/minicluster-data/ts-0/wal/wals/9a096516e01d44b8944b6221b94df89e
I20260620 14:50:41.506398 14793 ts_tablet_manager.cc:1939] T 9a096516e01d44b8944b6221b94df89e P cadfcb134c47484cae04962cad62ee16: Deleting consensus metadata
I20260620 14:50:41.507949 14794 ts_tablet_manager.cc:1905] T 9a096516e01d44b8944b6221b94df89e P 8fcc67d49a394238bc3c99b4560b78ec: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 14:50:41.508569 14278 catalog_manager.cc:4841] TS cadfcb134c47484cae04962cad62ee16 (127.0.97.1:42731): tablet 9a096516e01d44b8944b6221b94df89e (table test-table [id=48b19ad7987447e4b1ff739c5eeccc2d]) successfully deleted
I20260620 14:50:41.509594 14795 tablet_replica.cc:330] T 9a096516e01d44b8944b6221b94df89e P 1dcd6c66dba34248933d56d9ba564a97: stopping tablet replica
I20260620 14:50:41.510087 14795 raft_consensus.cc:2229] T 9a096516e01d44b8944b6221b94df89e P 1dcd6c66dba34248933d56d9ba564a97 [term 1 LEADER]: Raft consensus shutting down.
I20260620 14:50:41.510890 14795 raft_consensus.cc:2258] T 9a096516e01d44b8944b6221b94df89e P 1dcd6c66dba34248933d56d9ba564a97 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 14:50:41.513192 14795 ts_tablet_manager.cc:1905] T 9a096516e01d44b8944b6221b94df89e P 1dcd6c66dba34248933d56d9ba564a97: Deleting tablet data with delete state TABLET_DATA_DELETED
Jun 20 14:50:41 dist-test-slave-q9j6 krb5kdc[14249](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781967038, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.1@KRBTEST.COM
I20260620 14:50:41.516168 14794 ts_tablet_manager.cc:1918] T 9a096516e01d44b8944b6221b94df89e P 8fcc67d49a394238bc3c99b4560b78ec: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260620 14:50:41.516448 14794 log.cc:1196] T 9a096516e01d44b8944b6221b94df89e P 8fcc67d49a394238bc3c99b4560b78ec: Deleting WAL directory at /tmp/dist-test-taskUci3DU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781966829091300-388-0/minicluster-data/ts-2/wal/wals/9a096516e01d44b8944b6221b94df89e
I20260620 14:50:41.517001 14794 ts_tablet_manager.cc:1939] T 9a096516e01d44b8944b6221b94df89e P 8fcc67d49a394238bc3c99b4560b78ec: Deleting consensus metadata
I20260620 14:50:41.518999 14280 catalog_manager.cc:4841] TS 8fcc67d49a394238bc3c99b4560b78ec (127.0.97.3:35903): tablet 9a096516e01d44b8944b6221b94df89e (table test-table [id=48b19ad7987447e4b1ff739c5eeccc2d]) successfully deleted
I20260620 14:50:41.521103 14795 ts_tablet_manager.cc:1918] T 9a096516e01d44b8944b6221b94df89e P 1dcd6c66dba34248933d56d9ba564a97: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260620 14:50:41.521299 14795 log.cc:1196] T 9a096516e01d44b8944b6221b94df89e P 1dcd6c66dba34248933d56d9ba564a97: Deleting WAL directory at /tmp/dist-test-taskUci3DU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781966829091300-388-0/minicluster-data/ts-1/wal/wals/9a096516e01d44b8944b6221b94df89e
I20260620 14:50:41.521909 14795 ts_tablet_manager.cc:1939] T 9a096516e01d44b8944b6221b94df89e P 1dcd6c66dba34248933d56d9ba564a97: Deleting consensus metadata
I20260620 14:50:41.523777 14279 catalog_manager.cc:4841] TS 1dcd6c66dba34248933d56d9ba564a97 (127.0.97.2:45359): tablet 9a096516e01d44b8944b6221b94df89e (table test-table [id=48b19ad7987447e4b1ff739c5eeccc2d]) successfully deleted
I20260620 14:50:41.524938 14766 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:50:41.479561 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45896 (local address 127.0.97.1:42731)
0620 14:50:41.479837 (+ 276us) server_negotiation.cc:206] Beginning negotiation
0620 14:50:41.479846 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0620 14:50:41.503093 (+ 23247us) server_negotiation.cc:407] Connection header received
0620 14:50:41.503685 (+ 592us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:50:41.503694 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:50:41.503808 (+ 114us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:50:41.503895 (+ 87us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:50:41.506064 (+ 2169us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:50:41.507200 (+ 1136us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:50:41.510034 (+ 2834us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:50:41.510668 (+ 634us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:50:41.510788 (+ 120us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:50:41.517083 (+ 6295us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 14:50:41.517104 (+ 21us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 14:50:41.517126 (+ 22us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 14:50:41.517185 (+ 59us) server_negotiation.cc:922] calling sasl_server_start()
0620 14:50:41.521132 (+ 3947us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:50:41.522105 (+ 973us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:50:41.522117 (+ 12us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:50:41.522121 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:50:41.522303 (+ 182us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:50:41.522845 (+ 542us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:50:41.522851 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:50:41.522855 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:50:41.523346 (+ 491us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 14:50:41.524125 (+ 779us) server_negotiation.cc:1035] Waiting for connection context
0620 14:50:41.524315 (+ 190us) server_negotiation.cc:299] Negotiation successful
0620 14:50:41.524548 (+ 233us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":102}
I20260620 14:50:41.525653 14478 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 14:50:41.618783 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskUci3DU/build/asan/bin/kudu with pid 14355
I20260620 14:50:41.644243 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskUci3DU/build/asan/bin/kudu pid 14489
Jun 20 14:50:41 dist-test-slave-q9j6 krb5kdc[14249](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781967038, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.2@KRBTEST.COM
I20260620 14:50:41.662334 14769 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:50:41.645046 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49532 (local address 127.0.97.2:45359)
0620 14:50:41.645226 (+ 180us) server_negotiation.cc:206] Beginning negotiation
0620 14:50:41.645231 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 14:50:41.645309 (+ 78us) server_negotiation.cc:407] Connection header received
0620 14:50:41.645493 (+ 184us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:50:41.645498 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:50:41.645567 (+ 69us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:50:41.645665 (+ 98us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:50:41.647402 (+ 1737us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:50:41.648436 (+ 1034us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:50:41.650384 (+ 1948us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:50:41.650821 (+ 437us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:50:41.650913 (+ 92us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:50:41.656076 (+ 5163us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 14:50:41.656092 (+ 16us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 14:50:41.656109 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 14:50:41.656150 (+ 41us) server_negotiation.cc:922] calling sasl_server_start()
0620 14:50:41.659320 (+ 3170us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:50:41.660141 (+ 821us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:50:41.660149 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:50:41.660151 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:50:41.660302 (+ 151us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:50:41.660721 (+ 419us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:50:41.660733 (+ 12us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:50:41.660735 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:50:41.661096 (+ 361us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 14:50:41.661434 (+ 338us) server_negotiation.cc:1035] Waiting for connection context
0620 14:50:41.661734 (+ 300us) server_negotiation.cc:299] Negotiation successful
0620 14:50:41.661976 (+ 242us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":68}
I20260620 14:50:41.663097 14612 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 14:50:41.779801 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskUci3DU/build/asan/bin/kudu with pid 14489
I20260620 14:50:41.798566 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskUci3DU/build/asan/bin/kudu pid 14623
Jun 20 14:50:41 dist-test-slave-q9j6 krb5kdc[14249](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781967038, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.3@KRBTEST.COM
I20260620 14:50:41.814260 14767 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:50:41.799391 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45092 (local address 127.0.97.3:35903)
0620 14:50:41.799638 (+ 247us) server_negotiation.cc:206] Beginning negotiation
0620 14:50:41.799646 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0620 14:50:41.799677 (+ 31us) server_negotiation.cc:407] Connection header received
0620 14:50:41.799919 (+ 242us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:50:41.799923 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:50:41.800001 (+ 78us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:50:41.800087 (+ 86us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:50:41.801367 (+ 1280us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:50:41.802066 (+ 699us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:50:41.803299 (+ 1233us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:50:41.803845 (+ 546us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:50:41.803946 (+ 101us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:50:41.808436 (+ 4490us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 14:50:41.808453 (+ 17us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 14:50:41.808469 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 14:50:41.808507 (+ 38us) server_negotiation.cc:922] calling sasl_server_start()
0620 14:50:41.811494 (+ 2987us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:50:41.812252 (+ 758us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:50:41.812259 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:50:41.812261 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:50:41.812406 (+ 145us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:50:41.812756 (+ 350us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:50:41.812759 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:50:41.812761 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:50:41.813105 (+ 344us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 14:50:41.813324 (+ 219us) server_negotiation.cc:1035] Waiting for connection context
0620 14:50:41.813614 (+ 290us) server_negotiation.cc:299] Negotiation successful
0620 14:50:41.813781 (+ 167us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":73}
I20260620 14:50:41.814875 14746 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 14:50:41.904647 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskUci3DU/build/asan/bin/kudu with pid 14623
I20260620 14:50:41.921380 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskUci3DU/build/asan/bin/kudu pid 14262
I20260620 14:50:41.922605 14324 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 14:50:42.034968 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskUci3DU/build/asan/bin/kudu with pid 14262
I20260620 14:50:42.056911 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskUci3DU/build/asan/bin/kudu
/tmp/dist-test-taskUci3DU/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskUci3DU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781966829091300-388-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskUci3DU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781966829091300-388-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskUci3DU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781966829091300-388-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskUci3DU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781966829091300-388-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:33193
--webserver_interface=127.0.97.62
--webserver_port=45149
--keytab_file=/tmp/dist-test-taskUci3DU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781966829091300-388-0/krb5kdc/kudu_127.0.97.62.keytab
--principal=kudu/127.0.97.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:46747
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.97.62:33193
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--ipki_private_key_password_cmd=echo foo with env {KRB5CCNAME=/tmp/dist-test-taskUci3DU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781966829091300-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskUci3DU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781966829091300-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskUci3DU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781966829091300-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 14:50:42.400707 14801 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 14:50:42.401021 14801 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 14:50:42.401196 14801 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 14:50:42.409078 14801 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 14:50:42.409204 14801 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 14:50:42.409296 14801 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 14:50:42.409409 14801 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 14:50:42.409479 14801 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 14:50:42.419983 14801 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:46747
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskUci3DU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781966829091300-388-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskUci3DU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781966829091300-388-0/minicluster-data/master-0/wal
--ipki_private_key_password_cmd=echo foo
--ipki_ca_key_size=768
--master_addresses=127.0.97.62:33193
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskUci3DU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781966829091300-388-0/krb5kdc/kudu_127.0.97.62.keytab
--principal=kudu/127.0.97.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:33193
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskUci3DU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781966829091300-388-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.97.62
--webserver_port=45149
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskUci3DU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781966829091300-388-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 13:41:20 UTC on ccycloud.quasar-vdagoi.root.comops.site
build id 4457462
ASAN enabled
I20260620 14:50:42.421416 14801 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 14:50:42.423173 14801 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 14:50:42.447242 14807 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 14:50:42.447352 14808 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 14:50:42.447288 14810 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 14:50:42.448086 14801 server_base.cc:1034] running on GCE node
I20260620 14:50:42.449062 14801 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 14:50:42.450791 14801 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 14:50:42.452003 14801 hybrid_clock.cc:648] HybridClock initialized: now 1781967042451955 us; error 41 us; skew 500 ppm
I20260620 14:50:42.452507 14801 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 14:50:42 dist-test-slave-q9j6 krb5kdc[14249](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781967042, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 14:50:42.456233 14801 init.cc:377] Logged in from keytab as kudu/127.0.97.62@KRBTEST.COM (short username kudu)
I20260620 14:50:42.457978 14801 webserver.cc:460] Webserver started at http://127.0.97.62:45149/ using document root <none> and password file <none>
I20260620 14:50:42.458750 14801 fs_manager.cc:359] Metadata directory not provided
I20260620 14:50:42.458873 14801 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 14:50:42.465378 14801 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.004s sys 0.000s
I20260620 14:50:42.468699 14817 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 14:50:42.470247 14801 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.001s
I20260620 14:50:42.470464 14801 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskUci3DU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781966829091300-388-0/minicluster-data/master-0/data,/tmp/dist-test-taskUci3DU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781966829091300-388-0/minicluster-data/master-0/wal
uuid: "6629e744611e4470843f038539aebff5"
format_stamp: "Formatted at 2026-06-20 14:50:39 on dist-test-slave-q9j6"
server_key: "92fec065209290b45c346159a1a06aed"
server_key_iv: "3768cf0e4973befb91ba089285d6b9e3"
server_key_version: "encryptionkey@0"
I20260620 14:50:42.471751 14801 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskUci3DU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781966829091300-388-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskUci3DU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781966829091300-388-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskUci3DU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781966829091300-388-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 14:50:42.487668 14801 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 14:50:42.492609 14801 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 14:50:42.492941 14801 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 14:50:42.512732 14801 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.62:33193
I20260620 14:50:42.512712 14868 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.62:33193 every 8 connection(s)
I20260620 14:50:42.514432 14801 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskUci3DU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781966829091300-388-0/minicluster-data/master-0/data/info.pb
I20260620 14:50:42.521601 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskUci3DU/build/asan/bin/kudu as pid 14801
I20260620 14:50:42.522351 14869 sys_catalog.cc:263] Verifying existing consensus state
I20260620 14:50:42.522636 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskUci3DU/build/asan/bin/kudu
/tmp/dist-test-taskUci3DU/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskUci3DU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781966829091300-388-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskUci3DU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781966829091300-388-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskUci3DU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781966829091300-388-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskUci3DU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781966829091300-388-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.1:42731
--local_ip_for_outbound_sockets=127.0.97.1
--tserver_master_addrs=127.0.97.62:33193
--webserver_port=34811
--webserver_interface=127.0.97.1
--keytab_file=/tmp/dist-test-taskUci3DU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781966829091300-388-0/krb5kdc/kudu_127.0.97.1.keytab
--principal=kudu/127.0.97.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:46747
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskUci3DU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781966829091300-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskUci3DU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781966829091300-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskUci3DU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781966829091300-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260620 14:50:42.528106 14869 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 6629e744611e4470843f038539aebff5: Bootstrap starting.
I20260620 14:50:42.549705 14869 log.cc:824] T 00000000000000000000000000000000 P 6629e744611e4470843f038539aebff5: Log is configured to *not* fsync() on all Append() calls
I20260620 14:50:42.562769 14869 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 6629e744611e4470843f038539aebff5: Bootstrap replayed 1/1 log segments. Stats: ops{read=8 overwritten=0 applied=8 ignored=0} inserts{seen=5 ignored=0} mutations{seen=4 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260620 14:50:42.563370 14869 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 6629e744611e4470843f038539aebff5: Bootstrap complete.
I20260620 14:50:42.573176 14869 raft_consensus.cc:348] T 00000000000000000000000000000000 P 6629e744611e4470843f038539aebff5 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6629e744611e4470843f038539aebff5" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 33193 } }
I20260620 14:50:42.574143 14869 raft_consensus.cc:729] T 00000000000000000000000000000000 P 6629e744611e4470843f038539aebff5 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6629e744611e4470843f038539aebff5, State: Initialized, Role: FOLLOWER
I20260620 14:50:42.574880 14869 consensus_queue.cc:260] T 00000000000000000000000000000000 P 6629e744611e4470843f038539aebff5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 8, Last appended: 1.8, Last appended by leader: 8, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6629e744611e4470843f038539aebff5" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 33193 } }
I20260620 14:50:42.575150 14869 raft_consensus.cc:388] T 00000000000000000000000000000000 P 6629e744611e4470843f038539aebff5 [term 1 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 14:50:42.575305 14869 raft_consensus.cc:482] T 00000000000000000000000000000000 P 6629e744611e4470843f038539aebff5 [term 1 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 14:50:42.575513 14869 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 6629e744611e4470843f038539aebff5 [term 1 FOLLOWER]: Advancing to term 2
I20260620 14:50:42.579999 14869 raft_consensus.cc:504] T 00000000000000000000000000000000 P 6629e744611e4470843f038539aebff5 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6629e744611e4470843f038539aebff5" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 33193 } }
I20260620 14:50:42.580621 14869 leader_election.cc:302] T 00000000000000000000000000000000 P 6629e744611e4470843f038539aebff5 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 6629e744611e4470843f038539aebff5; no voters:
I20260620 14:50:42.581374 14869 leader_election.cc:288] T 00000000000000000000000000000000 P 6629e744611e4470843f038539aebff5 [CANDIDATE]: Term 2 election: Requested vote from peers
I20260620 14:50:42.581517 14873 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 6629e744611e4470843f038539aebff5 [term 2 FOLLOWER]: Leader election won for term 2
I20260620 14:50:42.582971 14873 raft_consensus.cc:686] T 00000000000000000000000000000000 P 6629e744611e4470843f038539aebff5 [term 2 LEADER]: Becoming Leader. State: Replica: 6629e744611e4470843f038539aebff5, State: Running, Role: LEADER
I20260620 14:50:42.583570 14873 consensus_queue.cc:237] T 00000000000000000000000000000000 P 6629e744611e4470843f038539aebff5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 1.8, Last appended by leader: 8, Current term: 2, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6629e744611e4470843f038539aebff5" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 33193 } }
I20260620 14:50:42.584194 14869 sys_catalog.cc:564] T 00000000000000000000000000000000 P 6629e744611e4470843f038539aebff5 [sys.catalog]: configured and running, proceeding with master startup.
I20260620 14:50:42.586350 14874 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6629e744611e4470843f038539aebff5 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 2 leader_uuid: "6629e744611e4470843f038539aebff5" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6629e744611e4470843f038539aebff5" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 33193 } } }
I20260620 14:50:42.586841 14874 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6629e744611e4470843f038539aebff5 [sys.catalog]: This master's current role is: LEADER
I20260620 14:50:42.587116 14875 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6629e744611e4470843f038539aebff5 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 6629e744611e4470843f038539aebff5. Latest consensus state: current_term: 2 leader_uuid: "6629e744611e4470843f038539aebff5" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6629e744611e4470843f038539aebff5" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 33193 } } }
I20260620 14:50:42.587579 14875 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6629e744611e4470843f038539aebff5 [sys.catalog]: This master's current role is: LEADER
I20260620 14:50:42.590009 14878 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 14:50:42.598697 14878 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 14:50:42.600327 14878 catalog_manager.cc:1259] Loaded cluster ID: 618885258ad7490b8093884920fa570e
I20260620 14:50:42.600857 14878 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 14:50:42.604079 14878 catalog_manager.cc:1314] IPKI private key was stored in cleartext, attempting to encrypt it.
I20260620 14:50:42.628851 14878 catalog_manager.cc:1324] IPKI private key successfully encrypted.
I20260620 14:50:42.647492 14878 catalog_manager.cc:1504] Loading token signing keys...
I20260620 14:50:42.649449 14878 catalog_manager.cc:5879] T 00000000000000000000000000000000 P 6629e744611e4470843f038539aebff5: Loaded TSK: 0
I20260620 14:50:42.651196 14878 catalog_manager.cc:1514] Initializing in-progress tserver states...
W20260620 14:50:42.901779 14871 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 14:50:42.902088 14871 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 14:50:42.902367 14871 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 14:50:42.910363 14871 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 14:50:42.910490 14871 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 14:50:42.910671 14871 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.1
I20260620 14:50:42.921223 14871 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:46747
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskUci3DU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781966829091300-388-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskUci3DU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781966829091300-388-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskUci3DU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781966829091300-388-0/krb5kdc/kudu_127.0.97.1.keytab
--principal=kudu/127.0.97.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.1:42731
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskUci3DU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781966829091300-388-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.97.1
--webserver_port=34811
--tserver_master_addrs=127.0.97.62:33193
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.1
--log_dir=/tmp/dist-test-taskUci3DU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781966829091300-388-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 13:41:20 UTC on ccycloud.quasar-vdagoi.root.comops.site
build id 4457462
ASAN enabled
I20260620 14:50:42.922667 14871 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 14:50:42.924151 14871 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 14:50:42.940074 14896 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 14:50:42.939960 14897 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 14:50:42.940876 14899 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 14:50:42.941501 14871 server_base.cc:1034] running on GCE node
I20260620 14:50:42.942269 14871 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 14:50:42.943362 14871 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 14:50:42.944618 14871 hybrid_clock.cc:648] HybridClock initialized: now 1781967042944540 us; error 71 us; skew 500 ppm
I20260620 14:50:42.944993 14871 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 14:50:42 dist-test-slave-q9j6 krb5kdc[14249](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781967042, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 14:50:42.952989 14871 init.cc:377] Logged in from keytab as kudu/127.0.97.1@KRBTEST.COM (short username kudu)
I20260620 14:50:42.954851 14871 webserver.cc:460] Webserver started at http://127.0.97.1:34811/ using document root <none> and password file <none>
I20260620 14:50:42.955686 14871 fs_manager.cc:359] Metadata directory not provided
I20260620 14:50:42.955845 14871 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 14:50:42.961719 14871 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.005s sys 0.000s
I20260620 14:50:42.965080 14906 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 14:50:42.966663 14871 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260620 14:50:42.966869 14871 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskUci3DU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781966829091300-388-0/minicluster-data/ts-0/data,/tmp/dist-test-taskUci3DU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781966829091300-388-0/minicluster-data/ts-0/wal
uuid: "cadfcb134c47484cae04962cad62ee16"
format_stamp: "Formatted at 2026-06-20 14:50:39 on dist-test-slave-q9j6"
server_key: "58d469cb46839aee2d40e48feaf4b64d"
server_key_iv: "eb6548832933138fc21456ab31548ac5"
server_key_version: "encryptionkey@0"
I20260620 14:50:42.967973 14871 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskUci3DU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781966829091300-388-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskUci3DU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781966829091300-388-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskUci3DU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781966829091300-388-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 14:50:42.982040 14871 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 14:50:42.986994 14871 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 14:50:42.987313 14871 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 14:50:42.988440 14871 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 14:50:42.989933 14871 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 14:50:42.990067 14871 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 14:50:42.990219 14871 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 14:50:42.990312 14871 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 14:50:43.024740 14871 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.1:42731
I20260620 14:50:43.024835 15018 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.1:42731 every 8 connection(s)
I20260620 14:50:43.026355 14871 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskUci3DU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781966829091300-388-0/minicluster-data/ts-0/data/info.pb
I20260620 14:50:43.030046 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskUci3DU/build/asan/bin/kudu as pid 14871
I20260620 14:50:43.030871 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskUci3DU/build/asan/bin/kudu
/tmp/dist-test-taskUci3DU/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskUci3DU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781966829091300-388-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskUci3DU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781966829091300-388-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskUci3DU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781966829091300-388-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskUci3DU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781966829091300-388-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.2:45359
--local_ip_for_outbound_sockets=127.0.97.2
--tserver_master_addrs=127.0.97.62:33193
--webserver_port=37687
--webserver_interface=127.0.97.2
--keytab_file=/tmp/dist-test-taskUci3DU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781966829091300-388-0/krb5kdc/kudu_127.0.97.2.keytab
--principal=kudu/127.0.97.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:46747
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskUci3DU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781966829091300-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskUci3DU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781966829091300-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskUci3DU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781966829091300-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Jun 20 14:50:43 dist-test-slave-q9j6 krb5kdc[14249](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781967042, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.1@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260620 14:50:43.056564 15022 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:50:43.029544 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:33193 (local address 127.0.97.1:45575)
0620 14:50:43.030641 (+ 1097us) negotiation.cc:107] Waiting for socket to connect
0620 14:50:43.030702 (+ 61us) client_negotiation.cc:174] Beginning negotiation
0620 14:50:43.033744 (+ 3042us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 14:50:43.035477 (+ 1733us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 14:50:43.035492 (+ 15us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 14:50:43.036382 (+ 890us) client_negotiation.cc:189] Negotiated authn=SASL
0620 14:50:43.037216 (+ 834us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:50:43.037246 (+ 30us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:50:43.039666 (+ 2420us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:50:43.039671 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:50:43.041076 (+ 1405us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:50:43.041091 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:50:43.042807 (+ 1716us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:50:43.042812 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:50:43.043030 (+ 218us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:50:43.043834 (+ 804us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 14:50:43.043862 (+ 28us) client_negotiation.cc:656] calling sasl_client_start()
0620 14:50:43.046479 (+ 2617us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 14:50:43.051859 (+ 5380us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 14:50:43.051867 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 14:50:43.051880 (+ 13us) client_negotiation.cc:760] calling sasl_client_step()
0620 14:50:43.052366 (+ 486us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 14:50:43.053008 (+ 642us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 14:50:43.053012 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 14:50:43.053014 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 14:50:43.053215 (+ 201us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 14:50:43.054368 (+ 1153us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 14:50:43.054375 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 14:50:43.054710 (+ 335us) client_negotiation.cc:769] Sending connection context
0620 14:50:43.054982 (+ 272us) client_negotiation.cc:240] Negotiation successful
0620 14:50:43.055411 (+ 429us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":758,"thread_start_us":500,"threads_started":1}
I20260620 14:50:43.057276 15021 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:50:43.029433 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.1:45575 (local address 127.0.97.62:33193)
0620 14:50:43.030242 (+ 809us) server_negotiation.cc:206] Beginning negotiation
0620 14:50:43.030261 (+ 19us) server_negotiation.cc:399] Waiting for connection header
0620 14:50:43.030887 (+ 626us) server_negotiation.cc:407] Connection header received
0620 14:50:43.034433 (+ 3546us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:50:43.034448 (+ 15us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:50:43.034864 (+ 416us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:50:43.035211 (+ 347us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:50:43.037498 (+ 2287us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:50:43.039400 (+ 1902us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:50:43.041264 (+ 1864us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:50:43.042535 (+ 1271us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:50:43.042778 (+ 243us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:50:43.046879 (+ 4101us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 14:50:43.046894 (+ 15us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 14:50:43.046921 (+ 27us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 14:50:43.046979 (+ 58us) server_negotiation.cc:922] calling sasl_server_start()
0620 14:50:43.051655 (+ 4676us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:50:43.052564 (+ 909us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:50:43.052579 (+ 15us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:50:43.052590 (+ 11us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:50:43.052872 (+ 282us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:50:43.053360 (+ 488us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:50:43.053368 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:50:43.053375 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:50:43.054201 (+ 826us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 14:50:43.054439 (+ 238us) server_negotiation.cc:1035] Waiting for connection context
0620 14:50:43.055290 (+ 851us) server_negotiation.cc:299] Negotiation successful
0620 14:50:43.055852 (+ 562us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":509,"thread_start_us":314,"threads_started":1}
I20260620 14:50:43.060424 15019 heartbeater.cc:347] Connected to a master server at 127.0.97.62:33193
I20260620 14:50:43.060782 15019 heartbeater.cc:464] Registering TS with master...
I20260620 14:50:43.061667 15019 heartbeater.cc:511] Master 127.0.97.62:33193 requested a full tablet report, sending...
I20260620 14:50:43.064034 14834 ts_manager.cc:194] Registered new tserver with Master: cadfcb134c47484cae04962cad62ee16 (127.0.97.1:42731)
I20260620 14:50:43.065539 14834 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.97.1@KRBTEST.COM'} at 127.0.97.1:45575
W20260620 14:50:43.380350 15023 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 14:50:43.380647 15023 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 14:50:43.380812 15023 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 14:50:43.388489 15023 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 14:50:43.388615 15023 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 14:50:43.388792 15023 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.2
I20260620 14:50:43.398959 15023 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:46747
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskUci3DU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781966829091300-388-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskUci3DU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781966829091300-388-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskUci3DU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781966829091300-388-0/krb5kdc/kudu_127.0.97.2.keytab
--principal=kudu/127.0.97.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.2:45359
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskUci3DU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781966829091300-388-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.97.2
--webserver_port=37687
--tserver_master_addrs=127.0.97.62:33193
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.2
--log_dir=/tmp/dist-test-taskUci3DU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781966829091300-388-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 13:41:20 UTC on ccycloud.quasar-vdagoi.root.comops.site
build id 4457462
ASAN enabled
I20260620 14:50:43.400408 15023 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 14:50:43.401763 15023 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 14:50:43.415777 15031 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 14:50:43.415823 15029 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 14:50:43.415737 15028 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 14:50:43.417061 15023 server_base.cc:1034] running on GCE node
I20260620 14:50:43.417801 15023 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 14:50:43.418861 15023 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 14:50:43.420100 15023 hybrid_clock.cc:648] HybridClock initialized: now 1781967043420035 us; error 63 us; skew 500 ppm
I20260620 14:50:43.420472 15023 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 14:50:43 dist-test-slave-q9j6 krb5kdc[14249](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781967043, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 14:50:43.423786 15023 init.cc:377] Logged in from keytab as kudu/127.0.97.2@KRBTEST.COM (short username kudu)
I20260620 14:50:43.425369 15023 webserver.cc:460] Webserver started at http://127.0.97.2:37687/ using document root <none> and password file <none>
I20260620 14:50:43.426055 15023 fs_manager.cc:359] Metadata directory not provided
I20260620 14:50:43.426180 15023 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 14:50:43.431581 15023 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.003s sys 0.000s
I20260620 14:50:43.434386 15038 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 14:50:43.435889 15023 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.002s
I20260620 14:50:43.436079 15023 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskUci3DU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781966829091300-388-0/minicluster-data/ts-1/data,/tmp/dist-test-taskUci3DU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781966829091300-388-0/minicluster-data/ts-1/wal
uuid: "1dcd6c66dba34248933d56d9ba564a97"
format_stamp: "Formatted at 2026-06-20 14:50:40 on dist-test-slave-q9j6"
server_key: "41d9e2f20f0aecac40cb62e44dd073f3"
server_key_iv: "283ebcdfb4a4714b498de7161c900549"
server_key_version: "encryptionkey@0"
I20260620 14:50:43.437104 15023 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskUci3DU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781966829091300-388-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskUci3DU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781966829091300-388-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskUci3DU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781966829091300-388-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 14:50:43.469158 15023 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 14:50:43.473538 15023 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 14:50:43.473834 15023 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 14:50:43.474845 15023 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 14:50:43.476503 15023 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 14:50:43.476619 15023 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 14:50:43.476756 15023 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 14:50:43.476850 15023 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 14:50:43.510987 15023 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.2:45359
I20260620 14:50:43.511045 15150 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.2:45359 every 8 connection(s)
I20260620 14:50:43.512584 15023 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskUci3DU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781966829091300-388-0/minicluster-data/ts-1/data/info.pb
I20260620 14:50:43.518589 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskUci3DU/build/asan/bin/kudu as pid 15023
I20260620 14:50:43.519199 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskUci3DU/build/asan/bin/kudu
/tmp/dist-test-taskUci3DU/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskUci3DU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781966829091300-388-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskUci3DU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781966829091300-388-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskUci3DU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781966829091300-388-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskUci3DU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781966829091300-388-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.3:35903
--local_ip_for_outbound_sockets=127.0.97.3
--tserver_master_addrs=127.0.97.62:33193
--webserver_port=33203
--webserver_interface=127.0.97.3
--keytab_file=/tmp/dist-test-taskUci3DU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781966829091300-388-0/krb5kdc/kudu_127.0.97.3.keytab
--principal=kudu/127.0.97.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:46747
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskUci3DU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781966829091300-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskUci3DU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781966829091300-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskUci3DU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781966829091300-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Jun 20 14:50:43 dist-test-slave-q9j6 krb5kdc[14249](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781967043, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.2@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260620 14:50:43.539091 15021 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:50:43.515486 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.2:38717 (local address 127.0.97.62:33193)
0620 14:50:43.515728 (+ 242us) server_negotiation.cc:206] Beginning negotiation
0620 14:50:43.515736 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0620 14:50:43.516988 (+ 1252us) server_negotiation.cc:407] Connection header received
0620 14:50:43.519577 (+ 2589us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:50:43.519584 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:50:43.519713 (+ 129us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:50:43.519794 (+ 81us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:50:43.522108 (+ 2314us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:50:43.523337 (+ 1229us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:50:43.525747 (+ 2410us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:50:43.526550 (+ 803us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:50:43.526718 (+ 168us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:50:43.531432 (+ 4714us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 14:50:43.531442 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 14:50:43.531451 (+ 9us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 14:50:43.531501 (+ 50us) server_negotiation.cc:922] calling sasl_server_start()
0620 14:50:43.535271 (+ 3770us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:50:43.536282 (+ 1011us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:50:43.536290 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:50:43.536296 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:50:43.536475 (+ 179us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:50:43.536942 (+ 467us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:50:43.536950 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:50:43.536955 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:50:43.537399 (+ 444us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 14:50:43.537574 (+ 175us) server_negotiation.cc:1035] Waiting for connection context
0620 14:50:43.538314 (+ 740us) server_negotiation.cc:299] Negotiation successful
0620 14:50:43.538621 (+ 307us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":76}
I20260620 14:50:43.540009 15153 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:50:43.515973 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:33193 (local address 127.0.97.2:38717)
0620 14:50:43.516797 (+ 824us) negotiation.cc:107] Waiting for socket to connect
0620 14:50:43.516849 (+ 52us) client_negotiation.cc:174] Beginning negotiation
0620 14:50:43.519239 (+ 2390us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 14:50:43.520353 (+ 1114us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 14:50:43.520365 (+ 12us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 14:50:43.521087 (+ 722us) client_negotiation.cc:189] Negotiated authn=SASL
0620 14:50:43.521881 (+ 794us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:50:43.521907 (+ 26us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:50:43.523551 (+ 1644us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:50:43.523557 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:50:43.525559 (+ 2002us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:50:43.525575 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:50:43.526816 (+ 1241us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:50:43.526823 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:50:43.527140 (+ 317us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:50:43.528027 (+ 887us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 14:50:43.528063 (+ 36us) client_negotiation.cc:656] calling sasl_client_start()
0620 14:50:43.531136 (+ 3073us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 14:50:43.535544 (+ 4408us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 14:50:43.535552 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 14:50:43.535567 (+ 15us) client_negotiation.cc:760] calling sasl_client_step()
0620 14:50:43.536123 (+ 556us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 14:50:43.536599 (+ 476us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 14:50:43.536602 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 14:50:43.536605 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0620 14:50:43.536809 (+ 204us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 14:50:43.537532 (+ 723us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 14:50:43.537538 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 14:50:43.537895 (+ 357us) client_negotiation.cc:769] Sending connection context
0620 14:50:43.538134 (+ 239us) client_negotiation.cc:240] Negotiation successful
0620 14:50:43.538617 (+ 483us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":516,"thread_start_us":313,"threads_started":1}
I20260620 14:50:43.541697 15151 heartbeater.cc:347] Connected to a master server at 127.0.97.62:33193
I20260620 14:50:43.542035 15151 heartbeater.cc:464] Registering TS with master...
I20260620 14:50:43.542805 15151 heartbeater.cc:511] Master 127.0.97.62:33193 requested a full tablet report, sending...
I20260620 14:50:43.544411 14834 ts_manager.cc:194] Registered new tserver with Master: 1dcd6c66dba34248933d56d9ba564a97 (127.0.97.2:45359)
I20260620 14:50:43.545235 14834 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.97.2@KRBTEST.COM'} at 127.0.97.2:38717
W20260620 14:50:43.863581 15154 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 14:50:43.864002 15154 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 14:50:43.864233 15154 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 14:50:43.871860 15154 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 14:50:43.872021 15154 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 14:50:43.872210 15154 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.3
I20260620 14:50:43.882592 15154 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:46747
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskUci3DU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781966829091300-388-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskUci3DU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781966829091300-388-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskUci3DU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781966829091300-388-0/krb5kdc/kudu_127.0.97.3.keytab
--principal=kudu/127.0.97.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.3:35903
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskUci3DU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781966829091300-388-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.97.3
--webserver_port=33203
--tserver_master_addrs=127.0.97.62:33193
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.3
--log_dir=/tmp/dist-test-taskUci3DU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781966829091300-388-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 13:41:20 UTC on ccycloud.quasar-vdagoi.root.comops.site
build id 4457462
ASAN enabled
I20260620 14:50:43.884137 15154 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 14:50:43.885586 15154 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 14:50:43.898874 15160 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 14:50:43.899047 15162 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 14:50:43.899083 15159 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 14:50:43.900175 15154 server_base.cc:1034] running on GCE node
I20260620 14:50:43.900876 15154 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 14:50:43.901885 15154 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 14:50:43.903118 15154 hybrid_clock.cc:648] HybridClock initialized: now 1781967043903071 us; error 45 us; skew 500 ppm
I20260620 14:50:43.903456 15154 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 14:50:43 dist-test-slave-q9j6 krb5kdc[14249](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781967043, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 14:50:43.906900 15154 init.cc:377] Logged in from keytab as kudu/127.0.97.3@KRBTEST.COM (short username kudu)
I20260620 14:50:43.908598 15154 webserver.cc:460] Webserver started at http://127.0.97.3:33203/ using document root <none> and password file <none>
I20260620 14:50:43.909334 15154 fs_manager.cc:359] Metadata directory not provided
I20260620 14:50:43.909497 15154 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 14:50:43.915093 15154 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.005s sys 0.000s
I20260620 14:50:43.917893 15169 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 14:50:43.919267 15154 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.000s sys 0.002s
I20260620 14:50:43.919447 15154 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskUci3DU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781966829091300-388-0/minicluster-data/ts-2/data,/tmp/dist-test-taskUci3DU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781966829091300-388-0/minicluster-data/ts-2/wal
uuid: "8fcc67d49a394238bc3c99b4560b78ec"
format_stamp: "Formatted at 2026-06-20 14:50:40 on dist-test-slave-q9j6"
server_key: "9787487b583dea31aca007219ece3219"
server_key_iv: "046b47dfdcc448e584d92f28422c36cd"
server_key_version: "encryptionkey@0"
I20260620 14:50:43.920552 15154 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskUci3DU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781966829091300-388-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskUci3DU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781966829091300-388-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskUci3DU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781966829091300-388-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 14:50:43.947703 15154 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 14:50:43.952333 15154 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 14:50:43.952638 15154 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 14:50:43.953742 15154 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 14:50:43.955231 15154 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 14:50:43.955359 15154 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 14:50:43.955495 15154 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 14:50:43.955582 15154 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 14:50:43.990356 15154 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.3:35903
I20260620 14:50:43.990447 15281 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.3:35903 every 8 connection(s)
I20260620 14:50:43.992017 15154 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskUci3DU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781966829091300-388-0/minicluster-data/ts-2/data/info.pb
I20260620 14:50:43.995817 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskUci3DU/build/asan/bin/kudu as pid 15154
Jun 20 14:50:44 dist-test-slave-q9j6 krb5kdc[14249](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781967043, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.3@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260620 14:50:44.012122 15285 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:50:43.996787 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57774 (local address 127.0.97.62:33193)
0620 14:50:43.997318 (+ 531us) server_negotiation.cc:206] Beginning negotiation
0620 14:50:43.997325 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0620 14:50:43.997675 (+ 350us) server_negotiation.cc:407] Connection header received
0620 14:50:43.997829 (+ 154us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:50:43.997837 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:50:43.997929 (+ 92us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:50:43.998061 (+ 132us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:50:43.999783 (+ 1722us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:50:44.000904 (+ 1121us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:50:44.002320 (+ 1416us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:50:44.002952 (+ 632us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:50:44.003084 (+ 132us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:50:44.005849 (+ 2765us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 14:50:44.005856 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 14:50:44.005860 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 14:50:44.005892 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0620 14:50:44.008952 (+ 3060us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:50:44.009810 (+ 858us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:50:44.009816 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:50:44.009819 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:50:44.009978 (+ 159us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:50:44.010534 (+ 556us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:50:44.010542 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:50:44.010544 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:50:44.010858 (+ 314us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 14:50:44.011155 (+ 297us) server_negotiation.cc:1035] Waiting for connection context
0620 14:50:44.011467 (+ 312us) server_negotiation.cc:299] Negotiation successful
0620 14:50:44.011744 (+ 277us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":398,"spinlock_wait_cycles":4480,"thread_start_us":270,"threads_started":1}
I20260620 14:50:44.015445 15021 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:50:43.994835 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.3:59209 (local address 127.0.97.62:33193)
0620 14:50:43.995029 (+ 194us) server_negotiation.cc:206] Beginning negotiation
0620 14:50:43.995034 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 14:50:43.996398 (+ 1364us) server_negotiation.cc:407] Connection header received
0620 14:50:43.998898 (+ 2500us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:50:43.998903 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:50:43.998979 (+ 76us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:50:43.999105 (+ 126us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:50:44.001249 (+ 2144us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:50:44.001965 (+ 716us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:50:44.004278 (+ 2313us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:50:44.004703 (+ 425us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:50:44.004862 (+ 159us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:50:44.008868 (+ 4006us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 14:50:44.008872 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 14:50:44.008876 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 14:50:44.008932 (+ 56us) server_negotiation.cc:922] calling sasl_server_start()
0620 14:50:44.011691 (+ 2759us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:50:44.012714 (+ 1023us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:50:44.012724 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:50:44.012726 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:50:44.012887 (+ 161us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:50:44.013431 (+ 544us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:50:44.013434 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:50:44.013436 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:50:44.013812 (+ 376us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 14:50:44.013928 (+ 116us) server_negotiation.cc:1035] Waiting for connection context
0620 14:50:44.014931 (+ 1003us) server_negotiation.cc:299] Negotiation successful
0620 14:50:44.015190 (+ 259us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":75,"spinlock_wait_cycles":7040}
I20260620 14:50:44.016572 15284 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:50:43.995331 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:33193 (local address 127.0.97.3:59209)
0620 14:50:43.996223 (+ 892us) negotiation.cc:107] Waiting for socket to connect
0620 14:50:43.996272 (+ 49us) client_negotiation.cc:174] Beginning negotiation
0620 14:50:43.998606 (+ 2334us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 14:50:43.999378 (+ 772us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 14:50:43.999388 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 14:50:44.000184 (+ 796us) client_negotiation.cc:189] Negotiated authn=SASL
0620 14:50:44.001065 (+ 881us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:50:44.001102 (+ 37us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:50:44.002131 (+ 1029us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:50:44.002136 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:50:44.004120 (+ 1984us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:50:44.004135 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:50:44.004954 (+ 819us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:50:44.004962 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:50:44.005231 (+ 269us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:50:44.005931 (+ 700us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 14:50:44.005956 (+ 25us) client_negotiation.cc:656] calling sasl_client_start()
0620 14:50:44.008593 (+ 2637us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 14:50:44.011839 (+ 3246us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 14:50:44.011849 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 14:50:44.011869 (+ 20us) client_negotiation.cc:760] calling sasl_client_step()
0620 14:50:44.012536 (+ 667us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 14:50:44.013021 (+ 485us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 14:50:44.013026 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 14:50:44.013030 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0620 14:50:44.013272 (+ 242us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 14:50:44.013958 (+ 686us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 14:50:44.013967 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 14:50:44.014459 (+ 492us) client_negotiation.cc:769] Sending connection context
0620 14:50:44.014810 (+ 351us) client_negotiation.cc:240] Negotiation successful
0620 14:50:44.015299 (+ 489us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":571,"thread_start_us":381,"threads_started":1}
I20260620 14:50:44.018098 15282 heartbeater.cc:347] Connected to a master server at 127.0.97.62:33193
I20260620 14:50:44.018469 15282 heartbeater.cc:464] Registering TS with master...
I20260620 14:50:44.019260 15282 heartbeater.cc:511] Master 127.0.97.62:33193 requested a full tablet report, sending...
I20260620 14:50:44.020992 14834 ts_manager.cc:194] Registered new tserver with Master: 8fcc67d49a394238bc3c99b4560b78ec (127.0.97.3:35903)
I20260620 14:50:44.021865 14834 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.97.3@KRBTEST.COM'} at 127.0.97.3:59209
I20260620 14:50:44.027571 388 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260620 14:50:44.034663 15021 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:50:44.028589 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57782 (local address 127.0.97.62:33193)
0620 14:50:44.028767 (+ 178us) server_negotiation.cc:206] Beginning negotiation
0620 14:50:44.028771 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 14:50:44.029258 (+ 487us) server_negotiation.cc:407] Connection header received
0620 14:50:44.029509 (+ 251us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:50:44.029519 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:50:44.029660 (+ 141us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:50:44.029807 (+ 147us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 14:50:44.030248 (+ 441us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:50:44.030999 (+ 751us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:50:44.032521 (+ 1522us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:50:44.032938 (+ 417us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:50:44.033075 (+ 137us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:50:44.033596 (+ 521us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 14:50:44.034086 (+ 490us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0620 14:50:44.034262 (+ 176us) server_negotiation.cc:1035] Waiting for connection context
0620 14:50:44.034322 (+ 60us) server_negotiation.cc:299] Negotiation successful
0620 14:50:44.034410 (+ 88us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":56}
I20260620 14:50:44.035648 14834 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin'} at 127.0.0.1:57782:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260620 14:50:44.043394 14834 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260620 14:50:44.069587 15019 heartbeater.cc:503] Master 127.0.97.62:33193 was elected leader, sending a full tablet report...
I20260620 14:50:44.081713 15293 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:50:44.056086 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45104 (local address 127.0.97.3:35903)
0620 14:50:44.069271 (+ 13185us) server_negotiation.cc:206] Beginning negotiation
0620 14:50:44.069281 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0620 14:50:44.069318 (+ 37us) server_negotiation.cc:407] Connection header received
0620 14:50:44.069835 (+ 517us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:50:44.069850 (+ 15us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:50:44.070509 (+ 659us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:50:44.072469 (+ 1960us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 14:50:44.072783 (+ 314us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:50:44.075242 (+ 2459us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:50:44.079410 (+ 4168us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:50:44.080437 (+ 1027us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:50:44.080656 (+ 219us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:50:44.080960 (+ 304us) server_negotiation.cc:1035] Waiting for connection context
0620 14:50:44.081171 (+ 211us) server_negotiation.cc:299] Negotiation successful
0620 14:50:44.081358 (+ 187us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":12981,"thread_start_us":12828,"threads_started":1}
I20260620 14:50:44.082719 15294 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:50:44.055901 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.3:35903 (local address 127.0.0.1:45104)
0620 14:50:44.068331 (+ 12430us) negotiation.cc:107] Waiting for socket to connect
0620 14:50:44.068379 (+ 48us) client_negotiation.cc:174] Beginning negotiation
0620 14:50:44.068579 (+ 200us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 14:50:44.070899 (+ 2320us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 14:50:44.070911 (+ 12us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 14:50:44.070969 (+ 58us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 14:50:44.071621 (+ 652us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:50:44.071641 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:50:44.075435 (+ 3794us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:50:44.075441 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:50:44.079237 (+ 3796us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:50:44.079248 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:50:44.080743 (+ 1495us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:50:44.080748 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:50:44.080894 (+ 146us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:50:44.080939 (+ 45us) client_negotiation.cc:769] Sending connection context
0620 14:50:44.081093 (+ 154us) client_negotiation.cc:240] Negotiation successful
0620 14:50:44.082273 (+ 1180us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":12165,"mutex_wait_us":1,"thread_start_us":20708,"threads_started":1}
I20260620 14:50:44.086086 15295 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:50:44.056429 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45908 (local address 127.0.97.1:42731)
0620 14:50:44.067374 (+ 10945us) server_negotiation.cc:206] Beginning negotiation
0620 14:50:44.067388 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0620 14:50:44.076771 (+ 9383us) server_negotiation.cc:407] Connection header received
0620 14:50:44.076911 (+ 140us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:50:44.076922 (+ 11us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:50:44.077231 (+ 309us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:50:44.077453 (+ 222us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 14:50:44.078682 (+ 1229us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:50:44.080421 (+ 1739us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:50:44.083531 (+ 3110us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:50:44.084886 (+ 1355us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:50:44.085052 (+ 166us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:50:44.085318 (+ 266us) server_negotiation.cc:1035] Waiting for connection context
0620 14:50:44.085588 (+ 270us) server_negotiation.cc:299] Negotiation successful
0620 14:50:44.085761 (+ 173us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":10678,"thread_start_us":11173,"threads_started":1}
I20260620 14:50:44.086194 15296 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:50:44.056083 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:42731 (local address 127.0.0.1:45908)
0620 14:50:44.076198 (+ 20115us) negotiation.cc:107] Waiting for socket to connect
0620 14:50:44.076222 (+ 24us) client_negotiation.cc:174] Beginning negotiation
0620 14:50:44.076357 (+ 135us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 14:50:44.077745 (+ 1388us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 14:50:44.077750 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 14:50:44.077774 (+ 24us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 14:50:44.078479 (+ 705us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:50:44.078490 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:50:44.080589 (+ 2099us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:50:44.080596 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:50:44.083344 (+ 2748us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:50:44.083362 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:50:44.085128 (+ 1766us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:50:44.085134 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:50:44.085385 (+ 251us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:50:44.085418 (+ 33us) client_negotiation.cc:769] Sending connection context
0620 14:50:44.085571 (+ 153us) client_negotiation.cc:240] Negotiation successful
0620 14:50:44.085734 (+ 163us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":19938,"mutex_wait_us":77,"spinlock_wait_cycles":33152,"thread_start_us":11808,"threads_started":1}
I20260620 14:50:44.086357 15292 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:50:44.057382 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.2:45359 (local address 127.0.0.1:49540)
0620 14:50:44.076911 (+ 19529us) negotiation.cc:107] Waiting for socket to connect
0620 14:50:44.076939 (+ 28us) client_negotiation.cc:174] Beginning negotiation
0620 14:50:44.077156 (+ 217us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 14:50:44.078074 (+ 918us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 14:50:44.078082 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 14:50:44.078118 (+ 36us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 14:50:44.078617 (+ 499us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:50:44.078630 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:50:44.081043 (+ 2413us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:50:44.081049 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:50:44.083344 (+ 2295us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:50:44.083362 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:50:44.085478 (+ 2116us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:50:44.085483 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:50:44.085655 (+ 172us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:50:44.085703 (+ 48us) client_negotiation.cc:769] Sending connection context
0620 14:50:44.085792 (+ 89us) client_negotiation.cc:240] Negotiation successful
0620 14:50:44.085931 (+ 139us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":19387,"mutex_wait_us":26,"thread_start_us":18459,"threads_started":1}
I20260620 14:50:44.088053 15297 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:50:44.058008 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49540 (local address 127.0.97.2:45359)
0620 14:50:44.075470 (+ 17462us) server_negotiation.cc:206] Beginning negotiation
0620 14:50:44.075483 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0620 14:50:44.077041 (+ 1558us) server_negotiation.cc:407] Connection header received
0620 14:50:44.077357 (+ 316us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:50:44.077368 (+ 11us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:50:44.077609 (+ 241us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:50:44.077782 (+ 173us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 14:50:44.078844 (+ 1062us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:50:44.080421 (+ 1577us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:50:44.083572 (+ 3151us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:50:44.084974 (+ 1402us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:50:44.085116 (+ 142us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:50:44.085308 (+ 192us) server_negotiation.cc:1035] Waiting for connection context
0620 14:50:44.087507 (+ 2199us) server_negotiation.cc:299] Negotiation successful
0620 14:50:44.087752 (+ 245us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":17247,"thread_start_us":17469,"threads_started":1}
I20260620 14:50:44.090487 14954 tablet_service.cc:1467] Processing CreateTablet for tablet f010a8b197d945509db8c96f3db322ed (DEFAULT_TABLE table=test-table [id=e2b96657e037486b9bf9d7b0fbc51053]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 14:50:44.088752 15217 tablet_service.cc:1467] Processing CreateTablet for tablet f010a8b197d945509db8c96f3db322ed (DEFAULT_TABLE table=test-table [id=e2b96657e037486b9bf9d7b0fbc51053]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 14:50:44.092139 14954 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f010a8b197d945509db8c96f3db322ed. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 14:50:44.091784 15086 tablet_service.cc:1467] Processing CreateTablet for tablet f010a8b197d945509db8c96f3db322ed (DEFAULT_TABLE table=test-table [id=e2b96657e037486b9bf9d7b0fbc51053]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 14:50:44.092505 15217 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f010a8b197d945509db8c96f3db322ed. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 14:50:44.093078 15086 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f010a8b197d945509db8c96f3db322ed. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 14:50:44.105859 15298 tablet_bootstrap.cc:492] T f010a8b197d945509db8c96f3db322ed P 8fcc67d49a394238bc3c99b4560b78ec: Bootstrap starting.
I20260620 14:50:44.108672 15300 tablet_bootstrap.cc:492] T f010a8b197d945509db8c96f3db322ed P cadfcb134c47484cae04962cad62ee16: Bootstrap starting.
I20260620 14:50:44.109508 15299 tablet_bootstrap.cc:492] T f010a8b197d945509db8c96f3db322ed P 1dcd6c66dba34248933d56d9ba564a97: Bootstrap starting.
I20260620 14:50:44.110543 15298 tablet_bootstrap.cc:654] T f010a8b197d945509db8c96f3db322ed P 8fcc67d49a394238bc3c99b4560b78ec: Neither blocks nor log segments found. Creating new log.
I20260620 14:50:44.112008 15298 log.cc:824] T f010a8b197d945509db8c96f3db322ed P 8fcc67d49a394238bc3c99b4560b78ec: Log is configured to *not* fsync() on all Append() calls
I20260620 14:50:44.113297 15299 tablet_bootstrap.cc:654] T f010a8b197d945509db8c96f3db322ed P 1dcd6c66dba34248933d56d9ba564a97: Neither blocks nor log segments found. Creating new log.
I20260620 14:50:44.114189 15300 tablet_bootstrap.cc:654] T f010a8b197d945509db8c96f3db322ed P cadfcb134c47484cae04962cad62ee16: Neither blocks nor log segments found. Creating new log.
I20260620 14:50:44.114650 15299 log.cc:824] T f010a8b197d945509db8c96f3db322ed P 1dcd6c66dba34248933d56d9ba564a97: Log is configured to *not* fsync() on all Append() calls
I20260620 14:50:44.115253 15300 log.cc:824] T f010a8b197d945509db8c96f3db322ed P cadfcb134c47484cae04962cad62ee16: Log is configured to *not* fsync() on all Append() calls
I20260620 14:50:44.115590 15298 tablet_bootstrap.cc:492] T f010a8b197d945509db8c96f3db322ed P 8fcc67d49a394238bc3c99b4560b78ec: No bootstrap required, opened a new log
I20260620 14:50:44.116004 15298 ts_tablet_manager.cc:1397] T f010a8b197d945509db8c96f3db322ed P 8fcc67d49a394238bc3c99b4560b78ec: Time spent bootstrapping tablet: real 0.010s user 0.005s sys 0.003s
I20260620 14:50:44.117823 15299 tablet_bootstrap.cc:492] T f010a8b197d945509db8c96f3db322ed P 1dcd6c66dba34248933d56d9ba564a97: No bootstrap required, opened a new log
I20260620 14:50:44.118122 15299 ts_tablet_manager.cc:1397] T f010a8b197d945509db8c96f3db322ed P 1dcd6c66dba34248933d56d9ba564a97: Time spent bootstrapping tablet: real 0.009s user 0.006s sys 0.000s
I20260620 14:50:44.118523 15300 tablet_bootstrap.cc:492] T f010a8b197d945509db8c96f3db322ed P cadfcb134c47484cae04962cad62ee16: No bootstrap required, opened a new log
I20260620 14:50:44.118799 15300 ts_tablet_manager.cc:1397] T f010a8b197d945509db8c96f3db322ed P cadfcb134c47484cae04962cad62ee16: Time spent bootstrapping tablet: real 0.010s user 0.005s sys 0.000s
I20260620 14:50:44.124615 15298 raft_consensus.cc:348] T f010a8b197d945509db8c96f3db322ed P 8fcc67d49a394238bc3c99b4560b78ec [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8fcc67d49a394238bc3c99b4560b78ec" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 35903 } } peers { permanent_uuid: "cadfcb134c47484cae04962cad62ee16" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 42731 } } peers { permanent_uuid: "1dcd6c66dba34248933d56d9ba564a97" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 45359 } }
I20260620 14:50:44.125128 15298 raft_consensus.cc:374] T f010a8b197d945509db8c96f3db322ed P 8fcc67d49a394238bc3c99b4560b78ec [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 14:50:44.125253 15298 raft_consensus.cc:729] T f010a8b197d945509db8c96f3db322ed P 8fcc67d49a394238bc3c99b4560b78ec [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8fcc67d49a394238bc3c99b4560b78ec, State: Initialized, Role: FOLLOWER
I20260620 14:50:44.125917 15298 consensus_queue.cc:260] T f010a8b197d945509db8c96f3db322ed P 8fcc67d49a394238bc3c99b4560b78ec [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8fcc67d49a394238bc3c99b4560b78ec" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 35903 } } peers { permanent_uuid: "cadfcb134c47484cae04962cad62ee16" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 42731 } } peers { permanent_uuid: "1dcd6c66dba34248933d56d9ba564a97" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 45359 } }
I20260620 14:50:44.127091 15282 heartbeater.cc:503] Master 127.0.97.62:33193 was elected leader, sending a full tablet report...
I20260620 14:50:44.127727 15298 ts_tablet_manager.cc:1428] T f010a8b197d945509db8c96f3db322ed P 8fcc67d49a394238bc3c99b4560b78ec: Time spent starting tablet: real 0.012s user 0.007s sys 0.005s
I20260620 14:50:44.128878 15299 raft_consensus.cc:348] T f010a8b197d945509db8c96f3db322ed P 1dcd6c66dba34248933d56d9ba564a97 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8fcc67d49a394238bc3c99b4560b78ec" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 35903 } } peers { permanent_uuid: "cadfcb134c47484cae04962cad62ee16" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 42731 } } peers { permanent_uuid: "1dcd6c66dba34248933d56d9ba564a97" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 45359 } }
I20260620 14:50:44.129352 15299 raft_consensus.cc:374] T f010a8b197d945509db8c96f3db322ed P 1dcd6c66dba34248933d56d9ba564a97 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 14:50:44.129487 15299 raft_consensus.cc:729] T f010a8b197d945509db8c96f3db322ed P 1dcd6c66dba34248933d56d9ba564a97 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1dcd6c66dba34248933d56d9ba564a97, State: Initialized, Role: FOLLOWER
I20260620 14:50:44.130129 15299 consensus_queue.cc:260] T f010a8b197d945509db8c96f3db322ed P 1dcd6c66dba34248933d56d9ba564a97 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8fcc67d49a394238bc3c99b4560b78ec" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 35903 } } peers { permanent_uuid: "cadfcb134c47484cae04962cad62ee16" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 42731 } } peers { permanent_uuid: "1dcd6c66dba34248933d56d9ba564a97" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 45359 } }
I20260620 14:50:44.130965 15300 raft_consensus.cc:348] T f010a8b197d945509db8c96f3db322ed P cadfcb134c47484cae04962cad62ee16 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8fcc67d49a394238bc3c99b4560b78ec" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 35903 } } peers { permanent_uuid: "cadfcb134c47484cae04962cad62ee16" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 42731 } } peers { permanent_uuid: "1dcd6c66dba34248933d56d9ba564a97" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 45359 } }
I20260620 14:50:44.131307 15151 heartbeater.cc:503] Master 127.0.97.62:33193 was elected leader, sending a full tablet report...
I20260620 14:50:44.131506 15300 raft_consensus.cc:374] T f010a8b197d945509db8c96f3db322ed P cadfcb134c47484cae04962cad62ee16 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 14:50:44.131673 15300 raft_consensus.cc:729] T f010a8b197d945509db8c96f3db322ed P cadfcb134c47484cae04962cad62ee16 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: cadfcb134c47484cae04962cad62ee16, State: Initialized, Role: FOLLOWER
I20260620 14:50:44.132067 15299 ts_tablet_manager.cc:1428] T f010a8b197d945509db8c96f3db322ed P 1dcd6c66dba34248933d56d9ba564a97: Time spent starting tablet: real 0.014s user 0.015s sys 0.001s
I20260620 14:50:44.132333 15300 consensus_queue.cc:260] T f010a8b197d945509db8c96f3db322ed P cadfcb134c47484cae04962cad62ee16 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8fcc67d49a394238bc3c99b4560b78ec" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 35903 } } peers { permanent_uuid: "cadfcb134c47484cae04962cad62ee16" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 42731 } } peers { permanent_uuid: "1dcd6c66dba34248933d56d9ba564a97" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 45359 } }
I20260620 14:50:44.134394 15300 ts_tablet_manager.cc:1428] T f010a8b197d945509db8c96f3db322ed P cadfcb134c47484cae04962cad62ee16: Time spent starting tablet: real 0.015s user 0.015s sys 0.000s
W20260620 14:50:44.243877 15283 tablet.cc:2369] T f010a8b197d945509db8c96f3db322ed P 8fcc67d49a394238bc3c99b4560b78ec: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260620 14:50:44.265048 15152 tablet.cc:2369] T f010a8b197d945509db8c96f3db322ed P 1dcd6c66dba34248933d56d9ba564a97: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260620 14:50:44.278913 15020 tablet.cc:2369] T f010a8b197d945509db8c96f3db322ed P cadfcb134c47484cae04962cad62ee16: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260620 14:50:44.332058 15305 raft_consensus.cc:482] T f010a8b197d945509db8c96f3db322ed P 1dcd6c66dba34248933d56d9ba564a97 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260620 14:50:44.332376 15305 raft_consensus.cc:504] T f010a8b197d945509db8c96f3db322ed P 1dcd6c66dba34248933d56d9ba564a97 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8fcc67d49a394238bc3c99b4560b78ec" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 35903 } } peers { permanent_uuid: "cadfcb134c47484cae04962cad62ee16" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 42731 } } peers { permanent_uuid: "1dcd6c66dba34248933d56d9ba564a97" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 45359 } }
I20260620 14:50:44.334053 15305 leader_election.cc:288] T f010a8b197d945509db8c96f3db322ed P 1dcd6c66dba34248933d56d9ba564a97 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 8fcc67d49a394238bc3c99b4560b78ec (127.0.97.3:35903), cadfcb134c47484cae04962cad62ee16 (127.0.97.1:42731)
I20260620 14:50:44.341339 15295 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:50:44.334694 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.2:40995 (local address 127.0.97.1:42731)
0620 14:50:44.335312 (+ 618us) server_negotiation.cc:206] Beginning negotiation
0620 14:50:44.335319 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0620 14:50:44.335648 (+ 329us) server_negotiation.cc:407] Connection header received
0620 14:50:44.335886 (+ 238us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:50:44.335891 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:50:44.335956 (+ 65us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:50:44.336085 (+ 129us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 14:50:44.336701 (+ 616us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:50:44.337414 (+ 713us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:50:44.339273 (+ 1859us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:50:44.340183 (+ 910us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:50:44.340346 (+ 163us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:50:44.340484 (+ 138us) server_negotiation.cc:1035] Waiting for connection context
0620 14:50:44.340926 (+ 442us) server_negotiation.cc:299] Negotiation successful
0620 14:50:44.341057 (+ 131us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":453}
I20260620 14:50:44.341490 15308 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:50:44.334427 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:42731 (local address 127.0.97.2:40995)
0620 14:50:44.335467 (+ 1040us) negotiation.cc:107] Waiting for socket to connect
0620 14:50:44.335495 (+ 28us) client_negotiation.cc:174] Beginning negotiation
0620 14:50:44.335731 (+ 236us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 14:50:44.336097 (+ 366us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 14:50:44.336102 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 14:50:44.336128 (+ 26us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 14:50:44.336534 (+ 406us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:50:44.336545 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:50:44.337553 (+ 1008us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:50:44.337559 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:50:44.339091 (+ 1532us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:50:44.339105 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:50:44.340416 (+ 1311us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:50:44.340422 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:50:44.340695 (+ 273us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:50:44.340741 (+ 46us) client_negotiation.cc:769] Sending connection context
0620 14:50:44.340901 (+ 160us) client_negotiation.cc:240] Negotiation successful
0620 14:50:44.341049 (+ 148us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":847,"thread_start_us":1064,"threads_started":1}
I20260620 14:50:44.341512 15307 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:50:44.334488 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.3:35903 (local address 127.0.97.2:54185)
0620 14:50:44.335870 (+ 1382us) negotiation.cc:107] Waiting for socket to connect
0620 14:50:44.335893 (+ 23us) client_negotiation.cc:174] Beginning negotiation
0620 14:50:44.336031 (+ 138us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 14:50:44.336369 (+ 338us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 14:50:44.336373 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 14:50:44.336403 (+ 30us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 14:50:44.336751 (+ 348us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:50:44.336759 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:50:44.337777 (+ 1018us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:50:44.337782 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:50:44.339242 (+ 1460us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:50:44.339254 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:50:44.340580 (+ 1326us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:50:44.340587 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:50:44.340831 (+ 244us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:50:44.340855 (+ 24us) client_negotiation.cc:769] Sending connection context
0620 14:50:44.340960 (+ 105us) client_negotiation.cc:240] Negotiation successful
0620 14:50:44.341092 (+ 132us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1254,"thread_start_us":585,"threads_started":1}
I20260620 14:50:44.341421 15293 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:50:44.334663 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.2:54185 (local address 127.0.97.3:35903)
0620 14:50:44.334905 (+ 242us) server_negotiation.cc:206] Beginning negotiation
0620 14:50:44.334912 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0620 14:50:44.335986 (+ 1074us) server_negotiation.cc:407] Connection header received
0620 14:50:44.336177 (+ 191us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:50:44.336183 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:50:44.336245 (+ 62us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:50:44.336353 (+ 108us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 14:50:44.336918 (+ 565us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:50:44.337623 (+ 705us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:50:44.339459 (+ 1836us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:50:44.340332 (+ 873us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:50:44.340481 (+ 149us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:50:44.340594 (+ 113us) server_negotiation.cc:1035] Waiting for connection context
0620 14:50:44.341016 (+ 422us) server_negotiation.cc:299] Negotiation successful
0620 14:50:44.341148 (+ 132us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":74}
I20260620 14:50:44.342793 14974 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f010a8b197d945509db8c96f3db322ed" candidate_uuid: "1dcd6c66dba34248933d56d9ba564a97" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "cadfcb134c47484cae04962cad62ee16" is_pre_election: true
I20260620 14:50:44.342978 15237 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f010a8b197d945509db8c96f3db322ed" candidate_uuid: "1dcd6c66dba34248933d56d9ba564a97" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8fcc67d49a394238bc3c99b4560b78ec" is_pre_election: true
I20260620 14:50:44.343384 14974 raft_consensus.cc:2454] T f010a8b197d945509db8c96f3db322ed P cadfcb134c47484cae04962cad62ee16 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1dcd6c66dba34248933d56d9ba564a97 in term 0.
I20260620 14:50:44.343532 15237 raft_consensus.cc:2454] T f010a8b197d945509db8c96f3db322ed P 8fcc67d49a394238bc3c99b4560b78ec [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1dcd6c66dba34248933d56d9ba564a97 in term 0.
I20260620 14:50:44.344233 15039 leader_election.cc:302] T f010a8b197d945509db8c96f3db322ed P 1dcd6c66dba34248933d56d9ba564a97 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1dcd6c66dba34248933d56d9ba564a97, cadfcb134c47484cae04962cad62ee16; no voters:
I20260620 14:50:44.344651 15305 raft_consensus.cc:2781] T f010a8b197d945509db8c96f3db322ed P 1dcd6c66dba34248933d56d9ba564a97 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260620 14:50:44.344791 15305 raft_consensus.cc:482] T f010a8b197d945509db8c96f3db322ed P 1dcd6c66dba34248933d56d9ba564a97 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260620 14:50:44.344913 15305 raft_consensus.cc:3037] T f010a8b197d945509db8c96f3db322ed P 1dcd6c66dba34248933d56d9ba564a97 [term 0 FOLLOWER]: Advancing to term 1
I20260620 14:50:44.347095 15305 raft_consensus.cc:504] T f010a8b197d945509db8c96f3db322ed P 1dcd6c66dba34248933d56d9ba564a97 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8fcc67d49a394238bc3c99b4560b78ec" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 35903 } } peers { permanent_uuid: "cadfcb134c47484cae04962cad62ee16" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 42731 } } peers { permanent_uuid: "1dcd6c66dba34248933d56d9ba564a97" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 45359 } }
I20260620 14:50:44.347800 15305 leader_election.cc:288] T f010a8b197d945509db8c96f3db322ed P 1dcd6c66dba34248933d56d9ba564a97 [CANDIDATE]: Term 1 election: Requested vote from peers 8fcc67d49a394238bc3c99b4560b78ec (127.0.97.3:35903), cadfcb134c47484cae04962cad62ee16 (127.0.97.1:42731)
I20260620 14:50:44.348263 15237 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f010a8b197d945509db8c96f3db322ed" candidate_uuid: "1dcd6c66dba34248933d56d9ba564a97" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8fcc67d49a394238bc3c99b4560b78ec"
I20260620 14:50:44.348351 14974 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f010a8b197d945509db8c96f3db322ed" candidate_uuid: "1dcd6c66dba34248933d56d9ba564a97" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "cadfcb134c47484cae04962cad62ee16"
I20260620 14:50:44.348472 15237 raft_consensus.cc:3037] T f010a8b197d945509db8c96f3db322ed P 8fcc67d49a394238bc3c99b4560b78ec [term 0 FOLLOWER]: Advancing to term 1
I20260620 14:50:44.348588 14974 raft_consensus.cc:3037] T f010a8b197d945509db8c96f3db322ed P cadfcb134c47484cae04962cad62ee16 [term 0 FOLLOWER]: Advancing to term 1
I20260620 14:50:44.350710 15237 raft_consensus.cc:2454] T f010a8b197d945509db8c96f3db322ed P 8fcc67d49a394238bc3c99b4560b78ec [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1dcd6c66dba34248933d56d9ba564a97 in term 1.
I20260620 14:50:44.350813 14974 raft_consensus.cc:2454] T f010a8b197d945509db8c96f3db322ed P cadfcb134c47484cae04962cad62ee16 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1dcd6c66dba34248933d56d9ba564a97 in term 1.
I20260620 14:50:44.351255 15041 leader_election.cc:302] T f010a8b197d945509db8c96f3db322ed P 1dcd6c66dba34248933d56d9ba564a97 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 3 yes votes; 0 no votes. yes voters: 1dcd6c66dba34248933d56d9ba564a97, 8fcc67d49a394238bc3c99b4560b78ec, cadfcb134c47484cae04962cad62ee16; no voters:
I20260620 14:50:44.351666 15305 raft_consensus.cc:2781] T f010a8b197d945509db8c96f3db322ed P 1dcd6c66dba34248933d56d9ba564a97 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 14:50:44.352104 15305 raft_consensus.cc:686] T f010a8b197d945509db8c96f3db322ed P 1dcd6c66dba34248933d56d9ba564a97 [term 1 LEADER]: Becoming Leader. State: Replica: 1dcd6c66dba34248933d56d9ba564a97, State: Running, Role: LEADER
I20260620 14:50:44.352603 15305 consensus_queue.cc:237] T f010a8b197d945509db8c96f3db322ed P 1dcd6c66dba34248933d56d9ba564a97 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8fcc67d49a394238bc3c99b4560b78ec" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 35903 } } peers { permanent_uuid: "cadfcb134c47484cae04962cad62ee16" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 42731 } } peers { permanent_uuid: "1dcd6c66dba34248933d56d9ba564a97" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 45359 } }
I20260620 14:50:44.357482 14832 catalog_manager.cc:5495] T f010a8b197d945509db8c96f3db322ed P 1dcd6c66dba34248933d56d9ba564a97 reported cstate change: term changed from 0 to 1, leader changed from <none> to 1dcd6c66dba34248933d56d9ba564a97 (127.0.97.2). New cstate: current_term: 1 leader_uuid: "1dcd6c66dba34248933d56d9ba564a97" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8fcc67d49a394238bc3c99b4560b78ec" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 35903 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "cadfcb134c47484cae04962cad62ee16" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 42731 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "1dcd6c66dba34248933d56d9ba564a97" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 45359 } health_report { overall_health: HEALTHY } } }
I20260620 14:50:44.379774 15297 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:50:44.373977 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49554 (local address 127.0.97.2:45359)
0620 14:50:44.374200 (+ 223us) server_negotiation.cc:206] Beginning negotiation
0620 14:50:44.374206 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 14:50:44.374227 (+ 21us) server_negotiation.cc:407] Connection header received
0620 14:50:44.374465 (+ 238us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:50:44.374470 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:50:44.374549 (+ 79us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:50:44.374704 (+ 155us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 14:50:44.375079 (+ 375us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:50:44.375930 (+ 851us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:50:44.377426 (+ 1496us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:50:44.377834 (+ 408us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:50:44.377971 (+ 137us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:50:44.378579 (+ 608us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 14:50:44.379108 (+ 529us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0620 14:50:44.379282 (+ 174us) server_negotiation.cc:1035] Waiting for connection context
0620 14:50:44.379345 (+ 63us) server_negotiation.cc:299] Negotiation successful
0620 14:50:44.379431 (+ 86us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":65}
I20260620 14:50:44.388537 15237 raft_consensus.cc:1261] T f010a8b197d945509db8c96f3db322ed P 8fcc67d49a394238bc3c99b4560b78ec [term 1 FOLLOWER]: Refusing update from remote peer 1dcd6c66dba34248933d56d9ba564a97: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 14:50:44.388535 14974 raft_consensus.cc:1261] T f010a8b197d945509db8c96f3db322ed P cadfcb134c47484cae04962cad62ee16 [term 1 FOLLOWER]: Refusing update from remote peer 1dcd6c66dba34248933d56d9ba564a97: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 14:50:44.389688 15309 consensus_queue.cc:1035] T f010a8b197d945509db8c96f3db322ed P 1dcd6c66dba34248933d56d9ba564a97 [LEADER]: Connected to new peer: Peer: permanent_uuid: "8fcc67d49a394238bc3c99b4560b78ec" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 35903 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 14:50:44.390067 15305 consensus_queue.cc:1035] T f010a8b197d945509db8c96f3db322ed P 1dcd6c66dba34248933d56d9ba564a97 [LEADER]: Connected to new peer: Peer: permanent_uuid: "cadfcb134c47484cae04962cad62ee16" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 42731 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 14:50:44.417557 14832 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin'} at 127.0.0.1:57782:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 14:50:44.417861 14832 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin'} at 127.0.0.1:57782:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 14:50:44.421482 14832 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 6629e744611e4470843f038539aebff5: Sending DeleteTablet for 3 replicas of tablet f010a8b197d945509db8c96f3db322ed
I20260620 14:50:44.422676 14954 tablet_service.cc:1514] Processing DeleteTablet for tablet f010a8b197d945509db8c96f3db322ed with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 14:50:44 UTC) from {username='kudu', principal='kudu/127.0.97.62@KRBTEST.COM'} at 127.0.0.1:45908
I20260620 14:50:44.422669 15217 tablet_service.cc:1514] Processing DeleteTablet for tablet f010a8b197d945509db8c96f3db322ed with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 14:50:44 UTC) from {username='kudu', principal='kudu/127.0.97.62@KRBTEST.COM'} at 127.0.0.1:45104
I20260620 14:50:44.423403 15086 tablet_service.cc:1514] Processing DeleteTablet for tablet f010a8b197d945509db8c96f3db322ed with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 14:50:44 UTC) from {username='kudu', principal='kudu/127.0.97.62@KRBTEST.COM'} at 127.0.0.1:49540
I20260620 14:50:44.423578 15321 tablet_replica.cc:330] T f010a8b197d945509db8c96f3db322ed P cadfcb134c47484cae04962cad62ee16: stopping tablet replica
I20260620 14:50:44.424084 15321 raft_consensus.cc:2229] T f010a8b197d945509db8c96f3db322ed P cadfcb134c47484cae04962cad62ee16 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 14:50:44.424661 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskUci3DU/build/asan/bin/kudu pid 14871
I20260620 14:50:44.424774 15321 raft_consensus.cc:2258] T f010a8b197d945509db8c96f3db322ed P cadfcb134c47484cae04962cad62ee16 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 14:50:44.430928 15321 ts_tablet_manager.cc:1905] T f010a8b197d945509db8c96f3db322ed P cadfcb134c47484cae04962cad62ee16: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 14:50:44.434506 15323 tablet_replica.cc:330] T f010a8b197d945509db8c96f3db322ed P 1dcd6c66dba34248933d56d9ba564a97: stopping tablet replica
I20260620 14:50:44.435045 15323 raft_consensus.cc:2229] T f010a8b197d945509db8c96f3db322ed P 1dcd6c66dba34248933d56d9ba564a97 [term 1 LEADER]: Raft consensus shutting down.
I20260620 14:50:44.436030 15323 raft_consensus.cc:2258] T f010a8b197d945509db8c96f3db322ed P 1dcd6c66dba34248933d56d9ba564a97 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 14:50:44.438419 15323 ts_tablet_manager.cc:1905] T f010a8b197d945509db8c96f3db322ed P 1dcd6c66dba34248933d56d9ba564a97: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 14:50:44.441409 15322 tablet_replica.cc:330] T f010a8b197d945509db8c96f3db322ed P 8fcc67d49a394238bc3c99b4560b78ec: stopping tablet replica
I20260620 14:50:44.443501 15321 ts_tablet_manager.cc:1918] T f010a8b197d945509db8c96f3db322ed P cadfcb134c47484cae04962cad62ee16: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260620 14:50:44.443795 15321 log.cc:1196] T f010a8b197d945509db8c96f3db322ed P cadfcb134c47484cae04962cad62ee16: Deleting WAL directory at /tmp/dist-test-taskUci3DU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781966829091300-388-0/minicluster-data/ts-0/wal/wals/f010a8b197d945509db8c96f3db322ed
I20260620 14:50:44.444505 15321 ts_tablet_manager.cc:1939] T f010a8b197d945509db8c96f3db322ed P cadfcb134c47484cae04962cad62ee16: Deleting consensus metadata
I20260620 14:50:44.445469 15322 raft_consensus.cc:2229] T f010a8b197d945509db8c96f3db322ed P 8fcc67d49a394238bc3c99b4560b78ec [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 14:50:44.446054 15322 raft_consensus.cc:2258] T f010a8b197d945509db8c96f3db322ed P 8fcc67d49a394238bc3c99b4560b78ec [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 14:50:44.446831 14818 catalog_manager.cc:4841] TS cadfcb134c47484cae04962cad62ee16 (127.0.97.1:42731): tablet f010a8b197d945509db8c96f3db322ed (table test-table [id=e2b96657e037486b9bf9d7b0fbc51053]) successfully deleted
I20260620 14:50:44.448230 15295 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:50:44.425518 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45918 (local address 127.0.97.1:42731)
0620 14:50:44.425683 (+ 165us) server_negotiation.cc:206] Beginning negotiation
0620 14:50:44.425691 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0620 14:50:44.425963 (+ 272us) server_negotiation.cc:407] Connection header received
0620 14:50:44.426041 (+ 78us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:50:44.426048 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:50:44.426165 (+ 117us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:50:44.426286 (+ 121us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:50:44.428835 (+ 2549us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:50:44.430330 (+ 1495us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:50:44.432997 (+ 2667us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:50:44.433957 (+ 960us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:50:44.434107 (+ 150us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:50:44.438181 (+ 4074us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 14:50:44.438198 (+ 17us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 14:50:44.438215 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 14:50:44.438269 (+ 54us) server_negotiation.cc:922] calling sasl_server_start()
0620 14:50:44.442871 (+ 4602us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:50:44.444010 (+ 1139us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:50:44.444020 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:50:44.444026 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:50:44.444251 (+ 225us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:50:44.444738 (+ 487us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:50:44.444746 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:50:44.444753 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:50:44.445215 (+ 462us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 14:50:44.445571 (+ 356us) server_negotiation.cc:1035] Waiting for connection context
0620 14:50:44.447520 (+ 1949us) server_negotiation.cc:299] Negotiation successful
0620 14:50:44.447826 (+ 306us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":54}
I20260620 14:50:44.448522 15322 ts_tablet_manager.cc:1905] T f010a8b197d945509db8c96f3db322ed P 8fcc67d49a394238bc3c99b4560b78ec: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 14:50:44.449290 15323 ts_tablet_manager.cc:1918] T f010a8b197d945509db8c96f3db322ed P 1dcd6c66dba34248933d56d9ba564a97: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260620 14:50:44.449549 15323 log.cc:1196] T f010a8b197d945509db8c96f3db322ed P 1dcd6c66dba34248933d56d9ba564a97: Deleting WAL directory at /tmp/dist-test-taskUci3DU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781966829091300-388-0/minicluster-data/ts-1/wal/wals/f010a8b197d945509db8c96f3db322ed
I20260620 14:50:44.449678 15014 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 14:50:44.450237 15323 ts_tablet_manager.cc:1939] T f010a8b197d945509db8c96f3db322ed P 1dcd6c66dba34248933d56d9ba564a97: Deleting consensus metadata
I20260620 14:50:44.452510 14819 catalog_manager.cc:4841] TS 1dcd6c66dba34248933d56d9ba564a97 (127.0.97.2:45359): tablet f010a8b197d945509db8c96f3db322ed (table test-table [id=e2b96657e037486b9bf9d7b0fbc51053]) successfully deleted
I20260620 14:50:44.461006 15322 ts_tablet_manager.cc:1918] T f010a8b197d945509db8c96f3db322ed P 8fcc67d49a394238bc3c99b4560b78ec: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260620 14:50:44.461207 15322 log.cc:1196] T f010a8b197d945509db8c96f3db322ed P 8fcc67d49a394238bc3c99b4560b78ec: Deleting WAL directory at /tmp/dist-test-taskUci3DU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781966829091300-388-0/minicluster-data/ts-2/wal/wals/f010a8b197d945509db8c96f3db322ed
I20260620 14:50:44.461663 15322 ts_tablet_manager.cc:1939] T f010a8b197d945509db8c96f3db322ed P 8fcc67d49a394238bc3c99b4560b78ec: Deleting consensus metadata
I20260620 14:50:44.463264 14820 catalog_manager.cc:4841] TS 8fcc67d49a394238bc3c99b4560b78ec (127.0.97.3:35903): tablet f010a8b197d945509db8c96f3db322ed (table test-table [id=e2b96657e037486b9bf9d7b0fbc51053]) successfully deleted
I20260620 14:50:44.563086 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskUci3DU/build/asan/bin/kudu with pid 14871
I20260620 14:50:44.581512 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskUci3DU/build/asan/bin/kudu pid 15023
I20260620 14:50:44.598132 15297 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:50:44.582481 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49570 (local address 127.0.97.2:45359)
0620 14:50:44.582655 (+ 174us) server_negotiation.cc:206] Beginning negotiation
0620 14:50:44.582660 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 14:50:44.582679 (+ 19us) server_negotiation.cc:407] Connection header received
0620 14:50:44.582779 (+ 100us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:50:44.582783 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:50:44.582848 (+ 65us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:50:44.582939 (+ 91us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:50:44.584773 (+ 1834us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:50:44.585646 (+ 873us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:50:44.587744 (+ 2098us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:50:44.588204 (+ 460us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:50:44.588326 (+ 122us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:50:44.591831 (+ 3505us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 14:50:44.591846 (+ 15us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 14:50:44.591862 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 14:50:44.591902 (+ 40us) server_negotiation.cc:922] calling sasl_server_start()
0620 14:50:44.595099 (+ 3197us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:50:44.596129 (+ 1030us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:50:44.596136 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:50:44.596139 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:50:44.596318 (+ 179us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:50:44.596710 (+ 392us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:50:44.596713 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:50:44.596715 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:50:44.597042 (+ 327us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 14:50:44.597307 (+ 265us) server_negotiation.cc:1035] Waiting for connection context
0620 14:50:44.597612 (+ 305us) server_negotiation.cc:299] Negotiation successful
0620 14:50:44.597785 (+ 173us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":58}
I20260620 14:50:44.598874 15146 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 14:50:44.716539 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskUci3DU/build/asan/bin/kudu with pid 15023
I20260620 14:50:44.736325 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskUci3DU/build/asan/bin/kudu pid 15154
I20260620 14:50:44.750332 15293 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:50:44.737101 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45120 (local address 127.0.97.3:35903)
0620 14:50:44.737281 (+ 180us) server_negotiation.cc:206] Beginning negotiation
0620 14:50:44.737289 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0620 14:50:44.737389 (+ 100us) server_negotiation.cc:407] Connection header received
0620 14:50:44.737614 (+ 225us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:50:44.737618 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:50:44.737685 (+ 67us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:50:44.737794 (+ 109us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:50:44.739214 (+ 1420us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:50:44.740010 (+ 796us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:50:44.741363 (+ 1353us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:50:44.741860 (+ 497us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:50:44.742011 (+ 151us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:50:44.744773 (+ 2762us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 14:50:44.744787 (+ 14us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 14:50:44.744803 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 14:50:44.744848 (+ 45us) server_negotiation.cc:922] calling sasl_server_start()
0620 14:50:44.747464 (+ 2616us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:50:44.748246 (+ 782us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:50:44.748259 (+ 13us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:50:44.748262 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:50:44.748414 (+ 152us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:50:44.748793 (+ 379us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:50:44.748798 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:50:44.748801 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:50:44.749153 (+ 352us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 14:50:44.749467 (+ 314us) server_negotiation.cc:1035] Waiting for connection context
0620 14:50:44.749752 (+ 285us) server_negotiation.cc:299] Negotiation successful
0620 14:50:44.749971 (+ 219us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":62}
I20260620 14:50:44.750939 15277 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 14:50:44.841192 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskUci3DU/build/asan/bin/kudu with pid 15154
I20260620 14:50:44.858387 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskUci3DU/build/asan/bin/kudu pid 14801
I20260620 14:50:44.859649 14864 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 14:50:44.974659 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskUci3DU/build/asan/bin/kudu with pid 14801
I20260620 14:50:44.997010 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskUci3DU/build/asan/bin/kudu
/tmp/dist-test-taskUci3DU/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskUci3DU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781966829091300-388-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskUci3DU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781966829091300-388-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskUci3DU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781966829091300-388-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskUci3DU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781966829091300-388-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:33193
--webserver_interface=127.0.97.62
--webserver_port=45149
--keytab_file=/tmp/dist-test-taskUci3DU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781966829091300-388-0/krb5kdc/kudu_127.0.97.62.keytab
--principal=kudu/127.0.97.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:46747
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.97.62:33193
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskUci3DU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781966829091300-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskUci3DU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781966829091300-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskUci3DU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781966829091300-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 14:50:45.340781 15328 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 14:50:45.341090 15328 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 14:50:45.341223 15328 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 14:50:45.348879 15328 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 14:50:45.349005 15328 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 14:50:45.349104 15328 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 14:50:45.349221 15328 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 14:50:45.349288 15328 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 14:50:45.359699 15328 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:46747
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskUci3DU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781966829091300-388-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskUci3DU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781966829091300-388-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.97.62:33193
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskUci3DU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781966829091300-388-0/krb5kdc/kudu_127.0.97.62.keytab
--principal=kudu/127.0.97.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:33193
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskUci3DU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781966829091300-388-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.97.62
--webserver_port=45149
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskUci3DU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781966829091300-388-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 13:41:20 UTC on ccycloud.quasar-vdagoi.root.comops.site
build id 4457462
ASAN enabled
I20260620 14:50:45.361172 15328 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 14:50:45.362581 15328 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 14:50:45.374969 15333 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 14:50:45.374902 15336 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 14:50:45.374864 15334 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 14:50:45.376039 15328 server_base.cc:1034] running on GCE node
I20260620 14:50:45.376928 15328 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 14:50:45.378407 15328 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 14:50:45.379703 15328 hybrid_clock.cc:648] HybridClock initialized: now 1781967045379639 us; error 63 us; skew 500 ppm
I20260620 14:50:45.380088 15328 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 14:50:45 dist-test-slave-q9j6 krb5kdc[14249](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781967045, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 14:50:45.383257 15328 init.cc:377] Logged in from keytab as kudu/127.0.97.62@KRBTEST.COM (short username kudu)
I20260620 14:50:45.384931 15328 webserver.cc:460] Webserver started at http://127.0.97.62:45149/ using document root <none> and password file <none>
I20260620 14:50:45.385675 15328 fs_manager.cc:359] Metadata directory not provided
I20260620 14:50:45.385830 15328 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 14:50:45.391678 15328 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.000s sys 0.003s
I20260620 14:50:45.394690 15343 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 14:50:45.396245 15328 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.000s sys 0.003s
I20260620 14:50:45.396456 15328 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskUci3DU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781966829091300-388-0/minicluster-data/master-0/data,/tmp/dist-test-taskUci3DU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781966829091300-388-0/minicluster-data/master-0/wal
uuid: "6629e744611e4470843f038539aebff5"
format_stamp: "Formatted at 2026-06-20 14:50:39 on dist-test-slave-q9j6"
server_key: "92fec065209290b45c346159a1a06aed"
server_key_iv: "3768cf0e4973befb91ba089285d6b9e3"
server_key_version: "encryptionkey@0"
I20260620 14:50:45.397506 15328 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskUci3DU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781966829091300-388-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskUci3DU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781966829091300-388-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskUci3DU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781966829091300-388-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 14:50:45.422111 15328 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 14:50:45.426651 15328 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 14:50:45.426963 15328 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 14:50:45.450240 15328 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.62:33193
I20260620 14:50:45.450304 15394 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.62:33193 every 8 connection(s)
I20260620 14:50:45.452003 15328 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskUci3DU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781966829091300-388-0/minicluster-data/master-0/data/info.pb
I20260620 14:50:45.459265 15395 sys_catalog.cc:263] Verifying existing consensus state
I20260620 14:50:45.462169 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskUci3DU/build/asan/bin/kudu as pid 15328
I20260620 14:50:45.462762 15395 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 6629e744611e4470843f038539aebff5: Bootstrap starting.
I20260620 14:50:45.463058 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskUci3DU/build/asan/bin/kudu
/tmp/dist-test-taskUci3DU/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskUci3DU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781966829091300-388-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskUci3DU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781966829091300-388-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskUci3DU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781966829091300-388-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskUci3DU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781966829091300-388-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.1:42731
--local_ip_for_outbound_sockets=127.0.97.1
--tserver_master_addrs=127.0.97.62:33193
--webserver_port=34811
--webserver_interface=127.0.97.1
--keytab_file=/tmp/dist-test-taskUci3DU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781966829091300-388-0/krb5kdc/kudu_127.0.97.1.keytab
--principal=kudu/127.0.97.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:46747
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskUci3DU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781966829091300-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskUci3DU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781966829091300-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskUci3DU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781966829091300-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260620 14:50:45.483917 15395 log.cc:824] T 00000000000000000000000000000000 P 6629e744611e4470843f038539aebff5: Log is configured to *not* fsync() on all Append() calls
I20260620 14:50:45.502538 15395 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 6629e744611e4470843f038539aebff5: Bootstrap replayed 1/1 log segments. Stats: ops{read=14 overwritten=0 applied=14 ignored=0} inserts{seen=7 ignored=0} mutations{seen=9 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260620 14:50:45.503118 15395 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 6629e744611e4470843f038539aebff5: Bootstrap complete.
I20260620 14:50:45.512633 15395 raft_consensus.cc:348] T 00000000000000000000000000000000 P 6629e744611e4470843f038539aebff5 [term 2 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6629e744611e4470843f038539aebff5" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 33193 } }
I20260620 14:50:45.513576 15395 raft_consensus.cc:729] T 00000000000000000000000000000000 P 6629e744611e4470843f038539aebff5 [term 2 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6629e744611e4470843f038539aebff5, State: Initialized, Role: FOLLOWER
I20260620 14:50:45.514207 15395 consensus_queue.cc:260] T 00000000000000000000000000000000 P 6629e744611e4470843f038539aebff5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 14, Last appended: 2.14, Last appended by leader: 14, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6629e744611e4470843f038539aebff5" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 33193 } }
I20260620 14:50:45.514479 15395 raft_consensus.cc:388] T 00000000000000000000000000000000 P 6629e744611e4470843f038539aebff5 [term 2 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 14:50:45.514618 15395 raft_consensus.cc:482] T 00000000000000000000000000000000 P 6629e744611e4470843f038539aebff5 [term 2 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 14:50:45.514798 15395 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 6629e744611e4470843f038539aebff5 [term 2 FOLLOWER]: Advancing to term 3
I20260620 14:50:45.518757 15395 raft_consensus.cc:504] T 00000000000000000000000000000000 P 6629e744611e4470843f038539aebff5 [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6629e744611e4470843f038539aebff5" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 33193 } }
I20260620 14:50:45.519330 15395 leader_election.cc:302] T 00000000000000000000000000000000 P 6629e744611e4470843f038539aebff5 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 6629e744611e4470843f038539aebff5; no voters:
I20260620 14:50:45.520080 15395 leader_election.cc:288] T 00000000000000000000000000000000 P 6629e744611e4470843f038539aebff5 [CANDIDATE]: Term 3 election: Requested vote from peers
I20260620 14:50:45.520241 15399 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 6629e744611e4470843f038539aebff5 [term 3 FOLLOWER]: Leader election won for term 3
I20260620 14:50:45.521343 15399 raft_consensus.cc:686] T 00000000000000000000000000000000 P 6629e744611e4470843f038539aebff5 [term 3 LEADER]: Becoming Leader. State: Replica: 6629e744611e4470843f038539aebff5, State: Running, Role: LEADER
I20260620 14:50:45.522008 15399 consensus_queue.cc:237] T 00000000000000000000000000000000 P 6629e744611e4470843f038539aebff5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 14, Committed index: 14, Last appended: 2.14, Last appended by leader: 14, Current term: 3, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6629e744611e4470843f038539aebff5" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 33193 } }
I20260620 14:50:45.522447 15395 sys_catalog.cc:564] T 00000000000000000000000000000000 P 6629e744611e4470843f038539aebff5 [sys.catalog]: configured and running, proceeding with master startup.
I20260620 14:50:45.524775 15401 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6629e744611e4470843f038539aebff5 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 6629e744611e4470843f038539aebff5. Latest consensus state: current_term: 3 leader_uuid: "6629e744611e4470843f038539aebff5" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6629e744611e4470843f038539aebff5" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 33193 } } }
I20260620 14:50:45.525199 15401 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6629e744611e4470843f038539aebff5 [sys.catalog]: This master's current role is: LEADER
I20260620 14:50:45.524904 15400 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6629e744611e4470843f038539aebff5 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 3 leader_uuid: "6629e744611e4470843f038539aebff5" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6629e744611e4470843f038539aebff5" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 33193 } } }
I20260620 14:50:45.525318 15400 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6629e744611e4470843f038539aebff5 [sys.catalog]: This master's current role is: LEADER
I20260620 14:50:45.526103 15406 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 14:50:45.539556 15406 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 14:50:45.542048 15406 catalog_manager.cc:1259] Loaded cluster ID: 618885258ad7490b8093884920fa570e
I20260620 14:50:45.542263 15406 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
F20260620 14:50:45.544970 15406 catalog_manager.cc:1466] Initializing Kudu internal certificate authority failed: Runtime error: unable to load data from memory: error:0D0680A8:asn1 encoding routines:ASN1_CHECK_TLEN:wrong tag:tasn_dec.c:1220 error:0D06C03A:asn1 encoding routines:ASN1_D2I_EX_PRIMITIVE:nested asn1 error:tasn_dec.c:788 error:0D08303A:asn1 encoding routines:ASN1_TEMPLATE_NOEXP_D2I:nested asn1 error:tasn_dec.c:720:Field=version, Type=RSA error:04093004:rsa routines:OLD_RSA_PRIV_DECODE:RSA lib:rsa_ameth.c:121 error:0D0680A8:asn1 encoding routines:ASN1_CHECK_TLEN:wrong tag:tasn_dec.c:1220 error:0D06C03A:asn1 encoding routines:ASN1_D2I_EX_PRIMITIVE:nested asn1 error:tasn_dec.c:788 error:0D08303A:asn1 encoding routines:ASN1_TEMPLATE_NOEXP_D2I:nested asn1 error:tasn_dec.c:720:Field=version, Type=PKCS8_PRIV_KEY_INFO
*** Check failure stack trace: ***
@ 0x7f9313b589ad google::LogMessage::Fail() at ??:0
@ 0x7f9313b5b88a google::LogMessage::SendToLog() at ??:0
@ 0x7f9313b584de google::LogMessage::Flush() at ??:0
@ 0x7f9313b59a19 google::LogMessageFatal::~LogMessageFatal() at ??:0
@ 0x7f932958b512 kudu::master::CatalogManager::PrepareForLeadershipTask()::$_6::operator()() at ??:0
@ 0x7f9329588976 kudu::master::CatalogManager::PrepareForLeadershipTask() at ??:0
@ 0x7f9329610610 kudu::master::CatalogManager::ElectedAsLeaderCb()::$_2::operator()() at ??:0
@ 0x7f9329610439 std::_Function_handler<>::_M_invoke() at ??:0
@ 0x9edb7d std::function<>::operator()() at ??:?
@ 0x7f93167a4345 kudu::ThreadPool::DispatchThread() at ??:0
@ 0x7f93167a791c kudu::ThreadPool::CreateThread()::$_2::operator()() at ??:0
@ 0x7f93167a7799 std::_Function_handler<>::_M_invoke() at ??:0
@ 0x9edb7d std::function<>::operator()() at ??:?
@ 0x7f9316782eaf kudu::Thread::SuperviseThread() at ??:0
@ 0x7f9319a5c6db start_thread at ??:0
@ 0x7f93129f261f clone at ??:0
W20260620 14:50:45.851382 15397 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 14:50:45.851807 15397 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 14:50:45.851985 15397 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 14:50:45.859572 15397 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 14:50:45.859812 15397 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 14:50:45.859997 15397 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.1
I20260620 14:50:45.870280 15397 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:46747
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskUci3DU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781966829091300-388-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskUci3DU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781966829091300-388-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskUci3DU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781966829091300-388-0/krb5kdc/kudu_127.0.97.1.keytab
--principal=kudu/127.0.97.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.1:42731
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskUci3DU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781966829091300-388-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.97.1
--webserver_port=34811
--tserver_master_addrs=127.0.97.62:33193
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.1
--log_dir=/tmp/dist-test-taskUci3DU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781966829091300-388-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 13:41:20 UTC on ccycloud.quasar-vdagoi.root.comops.site
build id 4457462
ASAN enabled
I20260620 14:50:45.871734 15397 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 14:50:45.873097 15397 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 14:50:45.886905 15437 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 14:50:45.887029 15439 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 14:50:45.887503 15436 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 14:50:45.887915 15397 server_base.cc:1034] running on GCE node
I20260620 14:50:45.888513 15397 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 14:50:45.889464 15397 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 14:50:45.890715 15397 hybrid_clock.cc:648] HybridClock initialized: now 1781967045890657 us; error 49 us; skew 500 ppm
I20260620 14:50:45.891068 15397 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 14:50:45 dist-test-slave-q9j6 krb5kdc[14249](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781967045, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 14:50:45.894366 15397 init.cc:377] Logged in from keytab as kudu/127.0.97.1@KRBTEST.COM (short username kudu)
I20260620 14:50:45.896066 15397 webserver.cc:460] Webserver started at http://127.0.97.1:34811/ using document root <none> and password file <none>
I20260620 14:50:45.896864 15397 fs_manager.cc:359] Metadata directory not provided
I20260620 14:50:45.896971 15397 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 14:50:45.902582 15397 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.005s sys 0.000s
I20260620 14:50:45.905488 15446 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 14:50:45.907004 15397 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260620 14:50:45.907186 15397 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskUci3DU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781966829091300-388-0/minicluster-data/ts-0/data,/tmp/dist-test-taskUci3DU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781966829091300-388-0/minicluster-data/ts-0/wal
uuid: "cadfcb134c47484cae04962cad62ee16"
format_stamp: "Formatted at 2026-06-20 14:50:39 on dist-test-slave-q9j6"
server_key: "58d469cb46839aee2d40e48feaf4b64d"
server_key_iv: "eb6548832933138fc21456ab31548ac5"
server_key_version: "encryptionkey@0"
I20260620 14:50:45.908320 15397 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskUci3DU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781966829091300-388-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskUci3DU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781966829091300-388-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskUci3DU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781966829091300-388-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 14:50:45.929977 15397 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 14:50:45.934226 15397 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 14:50:45.934535 15397 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 14:50:45.935588 15397 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 14:50:45.937189 15397 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 14:50:45.937312 15397 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 14:50:45.937453 15397 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 14:50:45.937551 15397 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 14:50:45.972379 15397 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.1:42731
I20260620 14:50:45.972481 15558 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.1:42731 every 8 connection(s)
I20260620 14:50:45.973956 15397 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskUci3DU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781966829091300-388-0/minicluster-data/ts-0/data/info.pb
I20260620 14:50:45.978176 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskUci3DU/build/asan/bin/kudu as pid 15397
I20260620 14:50:45.978981 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskUci3DU/build/asan/bin/kudu
/tmp/dist-test-taskUci3DU/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskUci3DU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781966829091300-388-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskUci3DU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781966829091300-388-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskUci3DU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781966829091300-388-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskUci3DU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781966829091300-388-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.2:45359
--local_ip_for_outbound_sockets=127.0.97.2
--tserver_master_addrs=127.0.97.62:33193
--webserver_port=37687
--webserver_interface=127.0.97.2
--keytab_file=/tmp/dist-test-taskUci3DU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781966829091300-388-0/krb5kdc/kudu_127.0.97.2.keytab
--principal=kudu/127.0.97.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:46747
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskUci3DU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781966829091300-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskUci3DU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781966829091300-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskUci3DU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781966829091300-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260620 14:50:45.979781 15561 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:50:45.977114 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:33193 (local address 127.0.97.1:60005)
0620 14:50:45.978067 (+ 953us) negotiation.cc:107] Waiting for socket to connect
0620 14:50:45.978373 (+ 306us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:33193: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":562,"thread_start_us":317,"threads_started":1}
W20260620 14:50:45.980535 15559 heartbeater.cc:644] Failed to heartbeat to 127.0.97.62:33193 (0 consecutive failures): Network error: Failed to ping master at 127.0.97.62:33193: Client connection negotiation failed: client connection to 127.0.97.62:33193: connect: Connection refused (error 111)
I20260620 14:50:45.981595 15561 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:50:45.981104 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:33193 (local address 127.0.97.1:39379)
0620 14:50:45.981323 (+ 219us) negotiation.cc:107] Waiting for socket to connect
0620 14:50:45.981431 (+ 108us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:33193: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":69}
I20260620 14:50:45.982681 15561 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:50:45.982224 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:33193 (local address 127.0.97.1:57481)
0620 14:50:45.982408 (+ 184us) negotiation.cc:107] Waiting for socket to connect
0620 14:50:45.982515 (+ 107us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:33193: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":58}
W20260620 14:50:45.982980 15559 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260620 14:50:46.323357 15562 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 14:50:46.323688 15562 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 14:50:46.323894 15562 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 14:50:46.331321 15562 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 14:50:46.331501 15562 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 14:50:46.331698 15562 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.2
I20260620 14:50:46.342344 15562 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:46747
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskUci3DU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781966829091300-388-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskUci3DU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781966829091300-388-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskUci3DU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781966829091300-388-0/krb5kdc/kudu_127.0.97.2.keytab
--principal=kudu/127.0.97.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.2:45359
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskUci3DU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781966829091300-388-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.97.2
--webserver_port=37687
--tserver_master_addrs=127.0.97.62:33193
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.2
--log_dir=/tmp/dist-test-taskUci3DU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781966829091300-388-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 13:41:20 UTC on ccycloud.quasar-vdagoi.root.comops.site
build id 4457462
ASAN enabled
I20260620 14:50:46.343688 15562 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 14:50:46.345072 15562 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 14:50:46.358937 15568 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 14:50:46.359128 15570 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 14:50:46.359083 15567 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 14:50:46.360121 15562 server_base.cc:1034] running on GCE node
I20260620 14:50:46.360772 15562 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 14:50:46.361768 15562 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 14:50:46.363016 15562 hybrid_clock.cc:648] HybridClock initialized: now 1781967046362938 us; error 71 us; skew 500 ppm
I20260620 14:50:46.363366 15562 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 14:50:46 dist-test-slave-q9j6 krb5kdc[14249](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781967046, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 14:50:46.366797 15562 init.cc:377] Logged in from keytab as kudu/127.0.97.2@KRBTEST.COM (short username kudu)
I20260620 14:50:46.368498 15562 webserver.cc:460] Webserver started at http://127.0.97.2:37687/ using document root <none> and password file <none>
I20260620 14:50:46.369241 15562 fs_manager.cc:359] Metadata directory not provided
I20260620 14:50:46.369396 15562 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 14:50:46.375341 15562 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.006s sys 0.000s
I20260620 14:50:46.378326 15577 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 14:50:46.379931 15562 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260620 14:50:46.380142 15562 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskUci3DU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781966829091300-388-0/minicluster-data/ts-1/data,/tmp/dist-test-taskUci3DU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781966829091300-388-0/minicluster-data/ts-1/wal
uuid: "1dcd6c66dba34248933d56d9ba564a97"
format_stamp: "Formatted at 2026-06-20 14:50:40 on dist-test-slave-q9j6"
server_key: "41d9e2f20f0aecac40cb62e44dd073f3"
server_key_iv: "283ebcdfb4a4714b498de7161c900549"
server_key_version: "encryptionkey@0"
I20260620 14:50:46.381562 15562 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskUci3DU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781966829091300-388-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskUci3DU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781966829091300-388-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskUci3DU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781966829091300-388-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 14:50:46.399973 15562 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 14:50:46.404840 15562 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 14:50:46.405135 15562 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 14:50:46.406169 15562 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 14:50:46.407651 15562 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 14:50:46.407788 15562 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 14:50:46.407948 15562 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 14:50:46.408017 15562 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 14:50:46.440925 15562 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.2:45359
I20260620 14:50:46.440979 15689 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.2:45359 every 8 connection(s)
I20260620 14:50:46.442524 15562 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskUci3DU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781966829091300-388-0/minicluster-data/ts-1/data/info.pb
I20260620 14:50:46.443919 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskUci3DU/build/asan/bin/kudu as pid 15562
I20260620 14:50:46.444626 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskUci3DU/build/asan/bin/kudu
/tmp/dist-test-taskUci3DU/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskUci3DU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781966829091300-388-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskUci3DU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781966829091300-388-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskUci3DU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781966829091300-388-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskUci3DU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781966829091300-388-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.3:35903
--local_ip_for_outbound_sockets=127.0.97.3
--tserver_master_addrs=127.0.97.62:33193
--webserver_port=33203
--webserver_interface=127.0.97.3
--keytab_file=/tmp/dist-test-taskUci3DU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781966829091300-388-0/krb5kdc/kudu_127.0.97.3.keytab
--principal=kudu/127.0.97.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:46747
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskUci3DU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781966829091300-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskUci3DU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781966829091300-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskUci3DU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781966829091300-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260620 14:50:46.448552 15692 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:50:46.445993 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:33193 (local address 127.0.97.2:59471)
0620 14:50:46.446932 (+ 939us) negotiation.cc:107] Waiting for socket to connect
0620 14:50:46.447206 (+ 274us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:33193: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":550,"thread_start_us":328,"threads_started":1}
W20260620 14:50:46.449416 15690 heartbeater.cc:644] Failed to heartbeat to 127.0.97.62:33193 (0 consecutive failures): Network error: Failed to ping master at 127.0.97.62:33193: Client connection negotiation failed: client connection to 127.0.97.62:33193: connect: Connection refused (error 111)
I20260620 14:50:46.450486 15692 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:50:46.449986 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:33193 (local address 127.0.97.2:49109)
0620 14:50:46.450207 (+ 221us) negotiation.cc:107] Waiting for socket to connect
0620 14:50:46.450315 (+ 108us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:33193: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":67}
I20260620 14:50:46.451514 15692 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:50:46.451168 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:33193 (local address 127.0.97.2:47499)
0620 14:50:46.451315 (+ 147us) negotiation.cc:107] Waiting for socket to connect
0620 14:50:46.451397 (+ 82us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:33193: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":58}
W20260620 14:50:46.451824 15690 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260620 14:50:46.807054 15693 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 14:50:46.807420 15693 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 14:50:46.807637 15693 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 14:50:46.816053 15693 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 14:50:46.816322 15693 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 14:50:46.816533 15693 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.3
I20260620 14:50:46.827293 15693 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:46747
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskUci3DU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781966829091300-388-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskUci3DU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781966829091300-388-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskUci3DU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781966829091300-388-0/krb5kdc/kudu_127.0.97.3.keytab
--principal=kudu/127.0.97.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.3:35903
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskUci3DU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781966829091300-388-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.97.3
--webserver_port=33203
--tserver_master_addrs=127.0.97.62:33193
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.3
--log_dir=/tmp/dist-test-taskUci3DU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781966829091300-388-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 13:41:20 UTC on ccycloud.quasar-vdagoi.root.comops.site
build id 4457462
ASAN enabled
I20260620 14:50:46.828897 15693 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 14:50:46.830288 15693 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 14:50:46.845045 15698 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 14:50:46.845011 15699 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 14:50:46.845011 15701 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 14:50:46.845997 15693 server_base.cc:1034] running on GCE node
I20260620 14:50:46.846984 15693 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 14:50:46.848130 15693 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 14:50:46.849380 15693 hybrid_clock.cc:648] HybridClock initialized: now 1781967046849326 us; error 51 us; skew 500 ppm
I20260620 14:50:46.849722 15693 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 14:50:46 dist-test-slave-q9j6 krb5kdc[14249](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781967046, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 14:50:46.853152 15693 init.cc:377] Logged in from keytab as kudu/127.0.97.3@KRBTEST.COM (short username kudu)
I20260620 14:50:46.854756 15693 webserver.cc:460] Webserver started at http://127.0.97.3:33203/ using document root <none> and password file <none>
I20260620 14:50:46.855521 15693 fs_manager.cc:359] Metadata directory not provided
I20260620 14:50:46.855706 15693 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 14:50:46.861774 15693 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.005s sys 0.000s
I20260620 14:50:46.864953 15708 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 14:50:46.866443 15693 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260620 14:50:46.866657 15693 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskUci3DU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781966829091300-388-0/minicluster-data/ts-2/data,/tmp/dist-test-taskUci3DU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781966829091300-388-0/minicluster-data/ts-2/wal
uuid: "8fcc67d49a394238bc3c99b4560b78ec"
format_stamp: "Formatted at 2026-06-20 14:50:40 on dist-test-slave-q9j6"
server_key: "9787487b583dea31aca007219ece3219"
server_key_iv: "046b47dfdcc448e584d92f28422c36cd"
server_key_version: "encryptionkey@0"
I20260620 14:50:46.867764 15693 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskUci3DU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781966829091300-388-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskUci3DU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781966829091300-388-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskUci3DU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781966829091300-388-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 14:50:46.890352 15693 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 14:50:46.894335 15693 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 14:50:46.894634 15693 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 14:50:46.895679 15693 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 14:50:46.897107 15693 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 14:50:46.897241 15693 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 14:50:46.897383 15693 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 14:50:46.897470 15693 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 14:50:46.930971 15693 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.3:35903
I20260620 14:50:46.931028 15820 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.3:35903 every 8 connection(s)
I20260620 14:50:46.932566 15693 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskUci3DU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781966829091300-388-0/minicluster-data/ts-2/data/info.pb
I20260620 14:50:46.937819 15823 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:50:46.935892 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:33193 (local address 127.0.97.3:57071)
0620 14:50:46.936646 (+ 754us) negotiation.cc:107] Waiting for socket to connect
0620 14:50:46.936880 (+ 234us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:33193: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":491,"thread_start_us":285,"threads_started":1}
W20260620 14:50:46.938529 15821 heartbeater.cc:644] Failed to heartbeat to 127.0.97.62:33193 (0 consecutive failures): Network error: Failed to ping master at 127.0.97.62:33193: Client connection negotiation failed: client connection to 127.0.97.62:33193: connect: Connection refused (error 111)
I20260620 14:50:46.939450 15823 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:50:46.939095 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:33193 (local address 127.0.97.3:38577)
0620 14:50:46.939264 (+ 169us) negotiation.cc:107] Waiting for socket to connect
0620 14:50:46.939345 (+ 81us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:33193: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":58}
I20260620 14:50:46.940395 15823 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:50:46.940039 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:33193 (local address 127.0.97.3:49679)
0620 14:50:46.940198 (+ 159us) negotiation.cc:107] Waiting for socket to connect
0620 14:50:46.940278 (+ 80us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:33193: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":56}
W20260620 14:50:46.940675 15821 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
I20260620 14:50:46.940976 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskUci3DU/build/asan/bin/kudu as pid 15693
I20260620 14:50:46.984388 15825 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:50:46.983540 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:33193 (local address 127.0.97.1:44799)
0620 14:50:46.984048 (+ 508us) negotiation.cc:107] Waiting for socket to connect
0620 14:50:46.984134 (+ 86us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:33193: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":401,"thread_start_us":276,"threads_started":1}
I20260620 14:50:47.453253 15826 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:50:47.452433 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:33193 (local address 127.0.97.2:38783)
0620 14:50:47.452895 (+ 462us) negotiation.cc:107] Waiting for socket to connect
0620 14:50:47.452984 (+ 89us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:33193: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":356,"thread_start_us":227,"threads_started":1}
I20260620 14:50:47.942065 15827 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:50:47.941235 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:33193 (local address 127.0.97.3:59319)
0620 14:50:47.941735 (+ 500us) negotiation.cc:107] Waiting for socket to connect
0620 14:50:47.941828 (+ 93us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:33193: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":384,"thread_start_us":252,"threads_started":1}
I20260620 14:50:47.986266 15828 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:50:47.985459 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:33193 (local address 127.0.97.1:52431)
0620 14:50:47.985919 (+ 460us) negotiation.cc:107] Waiting for socket to connect
0620 14:50:47.986007 (+ 88us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:33193: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":354,"thread_start_us":217,"threads_started":1}
I20260620 14:50:48.454988 15829 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:50:48.454244 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:33193 (local address 127.0.97.2:46229)
0620 14:50:48.454709 (+ 465us) negotiation.cc:107] Waiting for socket to connect
0620 14:50:48.454814 (+ 105us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:33193: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":353,"thread_start_us":225,"threads_started":1}
I20260620 14:50:48.943987 15830 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:50:48.943111 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:33193 (local address 127.0.97.3:36373)
0620 14:50:48.943701 (+ 590us) negotiation.cc:107] Waiting for socket to connect
0620 14:50:48.943800 (+ 99us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:33193: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":435,"thread_start_us":281,"threads_started":1}
I20260620 14:50:48.988144 15831 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:50:48.987347 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:33193 (local address 127.0.97.1:44761)
0620 14:50:48.987882 (+ 535us) negotiation.cc:107] Waiting for socket to connect
0620 14:50:48.987970 (+ 88us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:33193: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":430,"thread_start_us":309,"threads_started":1}
I20260620 14:50:49.456766 15832 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:50:49.456014 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:33193 (local address 127.0.97.2:38897)
0620 14:50:49.456503 (+ 489us) negotiation.cc:107] Waiting for socket to connect
0620 14:50:49.456590 (+ 87us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:33193: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":384,"thread_start_us":254,"threads_started":1}
I20260620 14:50:49.945811 15833 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:50:49.945062 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:33193 (local address 127.0.97.3:58549)
0620 14:50:49.945543 (+ 481us) negotiation.cc:107] Waiting for socket to connect
0620 14:50:49.945633 (+ 90us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:33193: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":364,"thread_start_us":248,"threads_started":1}
I20260620 14:50:49.989845 15834 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:50:49.989095 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:33193 (local address 127.0.97.1:56825)
0620 14:50:49.989581 (+ 486us) negotiation.cc:107] Waiting for socket to connect
0620 14:50:49.989667 (+ 86us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:33193: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":379,"thread_start_us":251,"threads_started":1}
I20260620 14:50:50.458350 15835 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:50:50.457684 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:33193 (local address 127.0.97.2:44055)
0620 14:50:50.458086 (+ 402us) negotiation.cc:107] Waiting for socket to connect
0620 14:50:50.458161 (+ 75us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:33193: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":308,"thread_start_us":199,"threads_started":1}
I20260620 14:50:50.947835 15836 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:50:50.946863 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:33193 (local address 127.0.97.3:42999)
0620 14:50:50.947428 (+ 565us) negotiation.cc:107] Waiting for socket to connect
0620 14:50:50.947532 (+ 104us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:33193: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":420,"thread_start_us":249,"threads_started":1}
I20260620 14:50:50.991571 15837 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:50:50.990835 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:33193 (local address 127.0.97.1:48563)
0620 14:50:50.991309 (+ 474us) negotiation.cc:107] Waiting for socket to connect
0620 14:50:50.991393 (+ 84us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:33193: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":373,"thread_start_us":241,"threads_started":1}
I20260620 14:50:51.460165 15838 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:50:51.459277 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:33193 (local address 127.0.97.2:47001)
0620 14:50:51.459842 (+ 565us) negotiation.cc:107] Waiting for socket to connect
0620 14:50:51.459965 (+ 123us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:33193: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":423,"thread_start_us":227,"threads_started":1}
I20260620 14:50:51.949756 15839 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:50:51.948941 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:33193 (local address 127.0.97.3:48047)
0620 14:50:51.949487 (+ 546us) negotiation.cc:107] Waiting for socket to connect
0620 14:50:51.949582 (+ 95us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:33193: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":437,"thread_start_us":275,"threads_started":1}
I20260620 14:50:51.993444 15840 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:50:51.992598 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:33193 (local address 127.0.97.1:48135)
0620 14:50:51.993099 (+ 501us) negotiation.cc:107] Waiting for socket to connect
0620 14:50:51.993259 (+ 160us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:33193: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":395,"thread_start_us":268,"threads_started":1}
I20260620 14:50:52.461812 15841 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:50:52.461101 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:33193 (local address 127.0.97.2:33075)
0620 14:50:52.461553 (+ 452us) negotiation.cc:107] Waiting for socket to connect
0620 14:50:52.461646 (+ 93us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:33193: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":352,"thread_start_us":237,"threads_started":1}
I20260620 14:50:52.951526 15842 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:50:52.950767 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:33193 (local address 127.0.97.3:39983)
0620 14:50:52.951229 (+ 462us) negotiation.cc:107] Waiting for socket to connect
0620 14:50:52.951358 (+ 129us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:33193: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":359,"thread_start_us":243,"threads_started":1}
I20260620 14:50:52.995249 15843 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:50:52.994456 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:33193 (local address 127.0.97.1:53343)
0620 14:50:52.994951 (+ 495us) negotiation.cc:107] Waiting for socket to connect
0620 14:50:52.995038 (+ 87us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:33193: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":387,"thread_start_us":216,"threads_started":1}
I20260620 14:50:53.463872 15844 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:50:53.462885 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:33193 (local address 127.0.97.2:49841)
0620 14:50:53.463463 (+ 578us) negotiation.cc:107] Waiting for socket to connect
0620 14:50:53.463549 (+ 86us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:33193: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":447,"thread_start_us":267,"threads_started":1}
I20260620 14:50:53.953423 15845 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:50:53.952588 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:33193 (local address 127.0.97.3:43591)
0620 14:50:53.953109 (+ 521us) negotiation.cc:107] Waiting for socket to connect
0620 14:50:53.953200 (+ 91us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:33193: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":405,"thread_start_us":223,"threads_started":1}
I20260620 14:50:53.996935 15846 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:50:53.996247 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:33193 (local address 127.0.97.1:48109)
0620 14:50:53.996684 (+ 437us) negotiation.cc:107] Waiting for socket to connect
0620 14:50:53.996768 (+ 84us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:33193: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":340,"thread_start_us":225,"threads_started":1}
I20260620 14:50:54.465641 15847 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:50:54.464900 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:33193 (local address 127.0.97.2:49507)
0620 14:50:54.465370 (+ 470us) negotiation.cc:107] Waiting for socket to connect
0620 14:50:54.465462 (+ 92us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:33193: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":363,"thread_start_us":244,"threads_started":1}
I20260620 14:50:54.955161 15848 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:50:54.954438 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:33193 (local address 127.0.97.3:54117)
0620 14:50:54.954897 (+ 459us) negotiation.cc:107] Waiting for socket to connect
0620 14:50:54.954989 (+ 92us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:33193: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":357,"thread_start_us":236,"threads_started":1}
I20260620 14:50:54.998596 15849 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:50:54.997845 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:33193 (local address 127.0.97.1:52885)
0620 14:50:54.998318 (+ 473us) negotiation.cc:107] Waiting for socket to connect
0620 14:50:54.998418 (+ 100us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:33193: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":366,"thread_start_us":232,"threads_started":1}
I20260620 14:50:55.467470 15850 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:50:55.466673 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:33193 (local address 127.0.97.2:44553)
0620 14:50:55.467176 (+ 503us) negotiation.cc:107] Waiting for socket to connect
0620 14:50:55.467278 (+ 102us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:33193: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":384,"thread_start_us":232,"threads_started":1}
I20260620 14:50:55.957093 15851 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:50:55.956303 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:33193 (local address 127.0.97.3:49769)
0620 14:50:55.956816 (+ 513us) negotiation.cc:107] Waiting for socket to connect
0620 14:50:55.956904 (+ 88us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:33193: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":402,"thread_start_us":260,"threads_started":1}
I20260620 14:50:56.000502 15852 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:50:55.999649 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:33193 (local address 127.0.97.1:38955)
0620 14:50:56.000222 (+ 573us) negotiation.cc:107] Waiting for socket to connect
0620 14:50:56.000308 (+ 86us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:33193: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":448,"thread_start_us":341,"threads_started":1}
I20260620 14:50:56.469364 15853 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:50:56.468629 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:33193 (local address 127.0.97.2:46167)
0620 14:50:56.469102 (+ 473us) negotiation.cc:107] Waiting for socket to connect
0620 14:50:56.469192 (+ 90us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:33193: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":360,"thread_start_us":239,"threads_started":1}
I20260620 14:50:56.958822 15854 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:50:56.958129 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:33193 (local address 127.0.97.3:60891)
0620 14:50:56.958571 (+ 442us) negotiation.cc:107] Waiting for socket to connect
0620 14:50:56.958656 (+ 85us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:33193: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":346,"thread_start_us":242,"threads_started":1}
I20260620 14:50:57.002269 15855 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:50:57.001505 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:33193 (local address 127.0.97.1:41585)
0620 14:50:57.001995 (+ 490us) negotiation.cc:107] Waiting for socket to connect
0620 14:50:57.002088 (+ 93us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:33193: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":376,"thread_start_us":261,"threads_started":1}
I20260620 14:50:57.471097 15856 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:50:57.470400 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:33193 (local address 127.0.97.2:60971)
0620 14:50:57.470821 (+ 421us) negotiation.cc:107] Waiting for socket to connect
0620 14:50:57.470910 (+ 89us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:33193: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":324,"thread_start_us":204,"threads_started":1}
I20260620 14:50:57.960654 15857 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:50:57.959907 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:33193 (local address 127.0.97.3:50393)
0620 14:50:57.960377 (+ 470us) negotiation.cc:107] Waiting for socket to connect
0620 14:50:57.960474 (+ 97us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:33193: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":365,"thread_start_us":228,"threads_started":1}
I20260620 14:50:58.004335 15858 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:50:58.003511 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:33193 (local address 127.0.97.1:59067)
0620 14:50:58.004053 (+ 542us) negotiation.cc:107] Waiting for socket to connect
0620 14:50:58.004152 (+ 99us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:33193: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":424,"thread_start_us":306,"threads_started":1}
I20260620 14:50:58.472874 15859 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:50:58.472056 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:33193 (local address 127.0.97.2:60167)
0620 14:50:58.472631 (+ 575us) negotiation.cc:107] Waiting for socket to connect
0620 14:50:58.472714 (+ 83us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:33193: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":455,"thread_start_us":317,"threads_started":1}
I20260620 14:50:58.962386 15860 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:50:58.961612 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:33193 (local address 127.0.97.3:37217)
0620 14:50:58.962148 (+ 536us) negotiation.cc:107] Waiting for socket to connect
0620 14:50:58.962234 (+ 86us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:33193: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":417,"thread_start_us":299,"threads_started":1}
I20260620 14:50:59.006376 15861 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:50:59.005402 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:33193 (local address 127.0.97.1:55627)
0620 14:50:59.006001 (+ 599us) negotiation.cc:107] Waiting for socket to connect
0620 14:50:59.006133 (+ 132us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:33193: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":460,"thread_start_us":316,"threads_started":1}
I20260620 14:50:59.474788 15862 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:50:59.473849 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:33193 (local address 127.0.97.2:41571)
0620 14:50:59.474484 (+ 635us) negotiation.cc:107] Waiting for socket to connect
0620 14:50:59.474590 (+ 106us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:33193: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":491,"thread_start_us":265,"threads_started":1}
I20260620 14:50:59.964155 15863 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:50:59.963299 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:33193 (local address 127.0.97.3:38719)
0620 14:50:59.963872 (+ 573us) negotiation.cc:107] Waiting for socket to connect
0620 14:50:59.963961 (+ 89us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:33193: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":465,"thread_start_us":304,"threads_started":1}
I20260620 14:51:00.008279 15864 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:51:00.007347 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:33193 (local address 127.0.97.1:47427)
0620 14:51:00.008011 (+ 664us) negotiation.cc:107] Waiting for socket to connect
0620 14:51:00.008093 (+ 82us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:33193: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":530,"thread_start_us":356,"threads_started":1}
I20260620 14:51:00.476631 15865 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:51:00.475817 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:33193 (local address 127.0.97.2:51943)
0620 14:51:00.476266 (+ 449us) negotiation.cc:107] Waiting for socket to connect
0620 14:51:00.476351 (+ 85us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:33193: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":335,"thread_start_us":215,"threads_started":1}
I20260620 14:51:00.965631 15866 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:51:00.964975 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:33193 (local address 127.0.97.3:58611)
0620 14:51:00.965375 (+ 400us) negotiation.cc:107] Waiting for socket to connect
0620 14:51:00.965449 (+ 74us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:33193: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":306,"thread_start_us":204,"threads_started":1}
I20260620 14:51:01.010215 15867 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:51:01.009488 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:33193 (local address 127.0.97.1:47513)
0620 14:51:01.009937 (+ 449us) negotiation.cc:107] Waiting for socket to connect
0620 14:51:01.010028 (+ 91us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:33193: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":345,"thread_start_us":238,"threads_started":1}
I20260620 14:51:01.478551 15868 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:51:01.477820 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:33193 (local address 127.0.97.2:39435)
0620 14:51:01.478272 (+ 452us) negotiation.cc:107] Waiting for socket to connect
0620 14:51:01.478366 (+ 94us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:33193: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":342,"thread_start_us":234,"threads_started":1}
I20260620 14:51:01.943188 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskUci3DU/build/asan/bin/kudu pid 15397
I20260620 14:51:01.966228 15869 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:51:01.944649 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37750 (local address 127.0.97.1:42731)
0620 14:51:01.945328 (+ 679us) server_negotiation.cc:206] Beginning negotiation
0620 14:51:01.945343 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0620 14:51:01.945386 (+ 43us) server_negotiation.cc:407] Connection header received
0620 14:51:01.947055 (+ 1669us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:51:01.947065 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:51:01.947540 (+ 475us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:51:01.947967 (+ 427us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:51:01.949432 (+ 1465us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:51:01.950710 (+ 1278us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:51:01.952084 (+ 1374us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:51:01.952816 (+ 732us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:51:01.952970 (+ 154us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:51:01.959236 (+ 6266us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 14:51:01.959246 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 14:51:01.959265 (+ 19us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 14:51:01.959317 (+ 52us) server_negotiation.cc:922] calling sasl_server_start()
0620 14:51:01.962571 (+ 3254us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:51:01.963433 (+ 862us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:51:01.963441 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:51:01.963450 (+ 9us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:51:01.963723 (+ 273us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:51:01.964078 (+ 355us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:51:01.964082 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:51:01.964084 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:51:01.964687 (+ 603us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 14:51:01.964939 (+ 252us) server_negotiation.cc:1035] Waiting for connection context
0620 14:51:01.965335 (+ 396us) server_negotiation.cc:299] Negotiation successful
0620 14:51:01.965840 (+ 505us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":504,"thread_start_us":330,"threads_started":1}
I20260620 14:51:01.967567 15870 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:51:01.966860 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:33193 (local address 127.0.97.3:37767)
0620 14:51:01.967300 (+ 440us) negotiation.cc:107] Waiting for socket to connect
0620 14:51:01.967389 (+ 89us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:33193: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":332,"thread_start_us":230,"threads_started":1}
I20260620 14:51:01.968094 15554 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 14:51:02.054318 15872 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:51:02.053144 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:33193 (local address 127.0.97.1:58445)
0620 14:51:02.053775 (+ 631us) negotiation.cc:107] Waiting for socket to connect
0620 14:51:02.053992 (+ 217us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:33193: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":488,"thread_start_us":296,"threads_started":1}
I20260620 14:51:02.055790 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskUci3DU/build/asan/bin/kudu with pid 15397
I20260620 14:51:02.073264 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskUci3DU/build/asan/bin/kudu pid 15562
I20260620 14:51:02.092634 15873 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:51:02.074330 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42510 (local address 127.0.97.2:45359)
0620 14:51:02.074875 (+ 545us) server_negotiation.cc:206] Beginning negotiation
0620 14:51:02.074890 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0620 14:51:02.074928 (+ 38us) server_negotiation.cc:407] Connection header received
0620 14:51:02.076531 (+ 1603us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:51:02.076543 (+ 12us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:51:02.077008 (+ 465us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:51:02.077332 (+ 324us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:51:02.078703 (+ 1371us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:51:02.080243 (+ 1540us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:51:02.081637 (+ 1394us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:51:02.082407 (+ 770us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:51:02.082567 (+ 160us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:51:02.085668 (+ 3101us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 14:51:02.085677 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 14:51:02.085696 (+ 19us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 14:51:02.085744 (+ 48us) server_negotiation.cc:922] calling sasl_server_start()
0620 14:51:02.089202 (+ 3458us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:51:02.089962 (+ 760us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:51:02.089968 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:51:02.089976 (+ 8us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:51:02.090160 (+ 184us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:51:02.090555 (+ 395us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:51:02.090559 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:51:02.090561 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:51:02.091179 (+ 618us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 14:51:02.091478 (+ 299us) server_negotiation.cc:1035] Waiting for connection context
0620 14:51:02.091928 (+ 450us) server_negotiation.cc:299] Negotiation successful
0620 14:51:02.092338 (+ 410us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":358,"thread_start_us":254,"threads_started":1}
I20260620 14:51:02.094223 15685 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 14:51:02.179371 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskUci3DU/build/asan/bin/kudu with pid 15562
I20260620 14:51:02.198104 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskUci3DU/build/asan/bin/kudu pid 15693
I20260620 14:51:02.216449 15875 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:51:02.199286 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47912 (local address 127.0.97.3:35903)
0620 14:51:02.199976 (+ 690us) server_negotiation.cc:206] Beginning negotiation
0620 14:51:02.199992 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0620 14:51:02.200042 (+ 50us) server_negotiation.cc:407] Connection header received
0620 14:51:02.201619 (+ 1577us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:51:02.201628 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:51:02.202094 (+ 466us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:51:02.202469 (+ 375us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:51:02.203996 (+ 1527us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:51:02.205259 (+ 1263us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:51:02.206570 (+ 1311us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:51:02.207267 (+ 697us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:51:02.207473 (+ 206us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:51:02.210117 (+ 2644us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 14:51:02.210125 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 14:51:02.210140 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 14:51:02.210178 (+ 38us) server_negotiation.cc:922] calling sasl_server_start()
0620 14:51:02.213026 (+ 2848us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:51:02.213780 (+ 754us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:51:02.213787 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:51:02.213794 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:51:02.213975 (+ 181us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:51:02.214375 (+ 400us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:51:02.214384 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:51:02.214386 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:51:02.214921 (+ 535us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 14:51:02.215243 (+ 322us) server_negotiation.cc:1035] Waiting for connection context
0620 14:51:02.215696 (+ 453us) server_negotiation.cc:299] Negotiation successful
0620 14:51:02.216144 (+ 448us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":527,"thread_start_us":374,"threads_started":1}
I20260620 14:51:02.217887 15816 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 14:51:02.303395 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskUci3DU/build/asan/bin/kudu with pid 15693
2026-06-20T14:51:02Z chronyd exiting
[ OK ] SecurityITest.TestEnableAndDisableEncryptedIPKICert (24803 ms)
[ RUN ] SecurityITest.TestEnableAndDisableEncryptedTSK
Loading random data
Initializing database '/tmp/dist-test-taskUci3DU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781966829091300-388-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 20 14:51:02 dist-test-slave-q9j6 krb5kdc[15879](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 20 14:51:02 dist-test-slave-q9j6 krb5kdc[15879](info): set up 2 sockets
Jun 20 14:51:02 dist-test-slave-q9j6 krb5kdc[15879](info): commencing operation
krb5kdc: starting...
W20260620 14:51:03.412299 388 mini_kdc.cc:121] Time spent starting KDC: real 1.068s user 0.003s sys 0.023s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskUci3DU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781966829091300-388-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskUci3DU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781966829091300-388-0/krb5kdc/test-user.keytab.
Jun 20 14:51:03 dist-test-slave-q9j6 krb5kdc[15879](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781967063, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-20T14:51:03Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T14:51:03Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.97.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskUci3DU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781966829091300-388-0/krb5kdc/kudu_127.0.97.62.keytab.
Entry for principal kudu/127.0.97.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskUci3DU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781966829091300-388-0/krb5kdc/kudu_127.0.97.62.keytab.
I20260620 14:51:03.657418 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskUci3DU/build/asan/bin/kudu
/tmp/dist-test-taskUci3DU/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskUci3DU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781966829091300-388-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskUci3DU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781966829091300-388-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskUci3DU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781966829091300-388-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskUci3DU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781966829091300-388-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:45085
--webserver_interface=127.0.97.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskUci3DU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781966829091300-388-0/krb5kdc/kudu_127.0.97.62.keytab
--principal=kudu/127.0.97.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:44619
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.97.62:45085
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_rotation_seconds=20 with env {KRB5CCNAME=/tmp/dist-test-taskUci3DU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781966829091300-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskUci3DU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781966829091300-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskUci3DU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781966829091300-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 14:51:04.007037 15892 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 14:51:04.007431 15892 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 14:51:04.007606 15892 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 14:51:04.015223 15892 flags.cc:425] Enabled experimental flag: --authn_token_validity_seconds=20
W20260620 14:51:04.015378 15892 flags.cc:425] Enabled experimental flag: --authz_token_validity_seconds=20
W20260620 14:51:04.015480 15892 flags.cc:425] Enabled experimental flag: --tsk_rotation_seconds=20
W20260620 14:51:04.015579 15892 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 14:51:04.015713 15892 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 14:51:04.015839 15892 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 14:51:04.015929 15892 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 14:51:04.016016 15892 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 14:51:04.027081 15892 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:44619
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskUci3DU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781966829091300-388-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskUci3DU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781966829091300-388-0/minicluster-data/master-0/wal
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_rotation_seconds=20
--ipki_ca_key_size=768
--master_addresses=127.0.97.62:45085
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskUci3DU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781966829091300-388-0/krb5kdc/kudu_127.0.97.62.keytab
--principal=kudu/127.0.97.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:45085
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskUci3DU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781966829091300-388-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.97.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskUci3DU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781966829091300-388-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 13:41:20 UTC on ccycloud.quasar-vdagoi.root.comops.site
build id 4457462
ASAN enabled
I20260620 14:51:04.028576 15892 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 14:51:04.029950 15892 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 14:51:04.042086 15898 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 14:51:04.042291 15897 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 14:51:04.042726 15900 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 14:51:04.043030 15892 server_base.cc:1034] running on GCE node
I20260620 14:51:04.043948 15892 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 14:51:04.045426 15892 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 14:51:04.046650 15892 hybrid_clock.cc:648] HybridClock initialized: now 1781967064046566 us; error 57 us; skew 500 ppm
I20260620 14:51:04.047030 15892 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 14:51:04 dist-test-slave-q9j6 krb5kdc[15879](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781967064, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 14:51:04.050493 15892 init.cc:377] Logged in from keytab as kudu/127.0.97.62@KRBTEST.COM (short username kudu)
I20260620 14:51:04.052178 15892 webserver.cc:460] Webserver started at http://127.0.97.62:32859/ using document root <none> and password file <none>
I20260620 14:51:04.052969 15892 fs_manager.cc:359] Metadata directory not provided
I20260620 14:51:04.053112 15892 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 14:51:04.053448 15892 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 14:51:04.056658 15892 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskUci3DU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781966829091300-388-0/minicluster-data/master-0/data/instance:
uuid: "0069e3cceda94cd68d41c5833a5174c9"
format_stamp: "Formatted at 2026-06-20 14:51:04 on dist-test-slave-q9j6"
server_key: "8a231203b28462a6eed146cdec780c4f"
server_key_iv: "1bb0c6765e03213cde26645f51bf2628"
server_key_version: "encryptionkey@0"
I20260620 14:51:04.057502 15892 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskUci3DU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781966829091300-388-0/minicluster-data/master-0/wal/instance:
uuid: "0069e3cceda94cd68d41c5833a5174c9"
format_stamp: "Formatted at 2026-06-20 14:51:04 on dist-test-slave-q9j6"
server_key: "8a231203b28462a6eed146cdec780c4f"
server_key_iv: "1bb0c6765e03213cde26645f51bf2628"
server_key_version: "encryptionkey@0"
I20260620 14:51:04.063084 15892 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.000s
I20260620 14:51:04.067052 15907 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 14:51:04.068583 15892 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260620 14:51:04.068800 15892 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskUci3DU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781966829091300-388-0/minicluster-data/master-0/data,/tmp/dist-test-taskUci3DU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781966829091300-388-0/minicluster-data/master-0/wal
uuid: "0069e3cceda94cd68d41c5833a5174c9"
format_stamp: "Formatted at 2026-06-20 14:51:04 on dist-test-slave-q9j6"
server_key: "8a231203b28462a6eed146cdec780c4f"
server_key_iv: "1bb0c6765e03213cde26645f51bf2628"
server_key_version: "encryptionkey@0"
I20260620 14:51:04.069054 15892 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskUci3DU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781966829091300-388-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskUci3DU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781966829091300-388-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskUci3DU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781966829091300-388-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 14:51:04.092690 15892 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 14:51:04.097280 15892 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 14:51:04.097621 15892 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 14:51:04.116334 15892 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.62:45085
I20260620 14:51:04.116367 15958 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.62:45085 every 8 connection(s)
I20260620 14:51:04.117844 15892 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskUci3DU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781966829091300-388-0/minicluster-data/master-0/data/info.pb
I20260620 14:51:04.121928 15959 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 14:51:04.122040 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskUci3DU/build/asan/bin/kudu as pid 15892
I20260620 14:51:04.122256 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskUci3DU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781966829091300-388-0/minicluster-data/master-0/wal/instance
I20260620 14:51:04.122903 388 external_mini_cluster.cc:1428] Setting key a009382998ae488cc4fb6ce7c6522665
I20260620 14:51:04.136890 15959 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 0069e3cceda94cd68d41c5833a5174c9: Bootstrap starting.
Jun 20 14:51:04 dist-test-slave-q9j6 krb5kdc[15879](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781967063, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260620 14:51:04.140211 15959 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 0069e3cceda94cd68d41c5833a5174c9: Neither blocks nor log segments found. Creating new log.
I20260620 14:51:04.141677 15959 log.cc:824] T 00000000000000000000000000000000 P 0069e3cceda94cd68d41c5833a5174c9: Log is configured to *not* fsync() on all Append() calls
I20260620 14:51:04.145380 15959 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 0069e3cceda94cd68d41c5833a5174c9: No bootstrap required, opened a new log
I20260620 14:51:04.149363 15962 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:51:04.124939 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60212 (local address 127.0.97.62:45085)
0620 14:51:04.125782 (+ 843us) server_negotiation.cc:206] Beginning negotiation
0620 14:51:04.125799 (+ 17us) server_negotiation.cc:399] Waiting for connection header
0620 14:51:04.125850 (+ 51us) server_negotiation.cc:407] Connection header received
0620 14:51:04.128847 (+ 2997us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:51:04.128881 (+ 34us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:51:04.129494 (+ 613us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:51:04.129987 (+ 493us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:51:04.131303 (+ 1316us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:51:04.133423 (+ 2120us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:51:04.134897 (+ 1474us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:51:04.136023 (+ 1126us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:51:04.136209 (+ 186us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:51:04.140386 (+ 4177us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 14:51:04.140398 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 14:51:04.140418 (+ 20us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 14:51:04.140474 (+ 56us) server_negotiation.cc:922] calling sasl_server_start()
0620 14:51:04.144449 (+ 3975us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:51:04.145179 (+ 730us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:51:04.145188 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:51:04.145196 (+ 8us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:51:04.145423 (+ 227us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:51:04.145912 (+ 489us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:51:04.145918 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:51:04.145921 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:51:04.146633 (+ 712us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 14:51:04.146936 (+ 303us) server_negotiation.cc:1035] Waiting for connection context
0620 14:51:04.147314 (+ 378us) server_negotiation.cc:299] Negotiation successful
0620 14:51:04.147850 (+ 536us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":604,"thread_start_us":354,"threads_started":1}
I20260620 14:51:04.155045 15959 raft_consensus.cc:348] T 00000000000000000000000000000000 P 0069e3cceda94cd68d41c5833a5174c9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0069e3cceda94cd68d41c5833a5174c9" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 45085 } }
I20260620 14:51:04.155411 15959 raft_consensus.cc:374] T 00000000000000000000000000000000 P 0069e3cceda94cd68d41c5833a5174c9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 14:51:04.155489 15959 raft_consensus.cc:729] T 00000000000000000000000000000000 P 0069e3cceda94cd68d41c5833a5174c9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0069e3cceda94cd68d41c5833a5174c9, State: Initialized, Role: FOLLOWER
I20260620 14:51:04.156082 15959 consensus_queue.cc:260] T 00000000000000000000000000000000 P 0069e3cceda94cd68d41c5833a5174c9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0069e3cceda94cd68d41c5833a5174c9" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 45085 } }
I20260620 14:51:04.156288 15959 raft_consensus.cc:388] T 00000000000000000000000000000000 P 0069e3cceda94cd68d41c5833a5174c9 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 14:51:04.156388 15959 raft_consensus.cc:482] T 00000000000000000000000000000000 P 0069e3cceda94cd68d41c5833a5174c9 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 14:51:04.156520 15959 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 0069e3cceda94cd68d41c5833a5174c9 [term 0 FOLLOWER]: Advancing to term 1
I20260620 14:51:04.158928 15959 raft_consensus.cc:504] T 00000000000000000000000000000000 P 0069e3cceda94cd68d41c5833a5174c9 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0069e3cceda94cd68d41c5833a5174c9" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 45085 } }
I20260620 14:51:04.159425 15959 leader_election.cc:302] T 00000000000000000000000000000000 P 0069e3cceda94cd68d41c5833a5174c9 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 0069e3cceda94cd68d41c5833a5174c9; no voters:
I20260620 14:51:04.160022 15959 leader_election.cc:288] T 00000000000000000000000000000000 P 0069e3cceda94cd68d41c5833a5174c9 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260620 14:51:04.160187 15964 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 0069e3cceda94cd68d41c5833a5174c9 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 14:51:04.160849 15964 raft_consensus.cc:686] T 00000000000000000000000000000000 P 0069e3cceda94cd68d41c5833a5174c9 [term 1 LEADER]: Becoming Leader. State: Replica: 0069e3cceda94cd68d41c5833a5174c9, State: Running, Role: LEADER
I20260620 14:51:04.161386 15964 consensus_queue.cc:237] T 00000000000000000000000000000000 P 0069e3cceda94cd68d41c5833a5174c9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0069e3cceda94cd68d41c5833a5174c9" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 45085 } }
I20260620 14:51:04.162120 15959 sys_catalog.cc:564] T 00000000000000000000000000000000 P 0069e3cceda94cd68d41c5833a5174c9 [sys.catalog]: configured and running, proceeding with master startup.
I20260620 14:51:04.164399 15966 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0069e3cceda94cd68d41c5833a5174c9 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 0069e3cceda94cd68d41c5833a5174c9. Latest consensus state: current_term: 1 leader_uuid: "0069e3cceda94cd68d41c5833a5174c9" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0069e3cceda94cd68d41c5833a5174c9" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 45085 } } }
I20260620 14:51:04.164822 15966 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0069e3cceda94cd68d41c5833a5174c9 [sys.catalog]: This master's current role is: LEADER
I20260620 14:51:04.165349 15965 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0069e3cceda94cd68d41c5833a5174c9 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "0069e3cceda94cd68d41c5833a5174c9" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0069e3cceda94cd68d41c5833a5174c9" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 45085 } } }
I20260620 14:51:04.165683 15965 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0069e3cceda94cd68d41c5833a5174c9 [sys.catalog]: This master's current role is: LEADER
I20260620 14:51:04.165788 15973 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 14:51:04.175544 15973 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 14:51:04.185094 15973 catalog_manager.cc:1347] Generated new cluster ID: 85e8db040d0d4fe1baaac9f6b8446522
I20260620 14:51:04.185253 15973 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 14:51:04.216666 15973 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 14:51:04.218690 15973 catalog_manager.cc:1504] Loading token signing keys...
I20260620 14:51:04.231503 15973 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 0069e3cceda94cd68d41c5833a5174c9: Generated new TSK 0
I20260620 14:51:04.232642 15973 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.0.97.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskUci3DU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781966829091300-388-0/krb5kdc/kudu_127.0.97.1.keytab.
Entry for principal kudu/127.0.97.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskUci3DU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781966829091300-388-0/krb5kdc/kudu_127.0.97.1.keytab.
I20260620 14:51:04.293527 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskUci3DU/build/asan/bin/kudu
/tmp/dist-test-taskUci3DU/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskUci3DU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781966829091300-388-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskUci3DU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781966829091300-388-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskUci3DU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781966829091300-388-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskUci3DU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781966829091300-388-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.1:0
--local_ip_for_outbound_sockets=127.0.97.1
--webserver_interface=127.0.97.1
--webserver_port=0
--tserver_master_addrs=127.0.97.62:45085
--keytab_file=/tmp/dist-test-taskUci3DU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781966829091300-388-0/krb5kdc/kudu_127.0.97.1.keytab
--principal=kudu/127.0.97.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:44619
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskUci3DU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781966829091300-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskUci3DU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781966829091300-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskUci3DU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781966829091300-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 14:51:04.652798 15985 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 14:51:04.653084 15985 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 14:51:04.653267 15985 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 14:51:04.660893 15985 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 14:51:04.661027 15985 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 14:51:04.661226 15985 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.1
I20260620 14:51:04.671800 15985 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:44619
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskUci3DU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781966829091300-388-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskUci3DU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781966829091300-388-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskUci3DU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781966829091300-388-0/krb5kdc/kudu_127.0.97.1.keytab
--principal=kudu/127.0.97.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskUci3DU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781966829091300-388-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.97.1
--webserver_port=0
--tserver_master_addrs=127.0.97.62:45085
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.1
--log_dir=/tmp/dist-test-taskUci3DU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781966829091300-388-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 13:41:20 UTC on ccycloud.quasar-vdagoi.root.comops.site
build id 4457462
ASAN enabled
I20260620 14:51:04.673249 15985 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 14:51:04.674690 15985 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 14:51:04.688895 15990 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 14:51:04.688859 15991 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 14:51:04.688935 15993 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 14:51:04.689785 15985 server_base.cc:1034] running on GCE node
I20260620 14:51:04.690428 15985 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 14:51:04.691427 15985 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 14:51:04.692659 15985 hybrid_clock.cc:648] HybridClock initialized: now 1781967064692608 us; error 71 us; skew 500 ppm
I20260620 14:51:04.692993 15985 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 14:51:04 dist-test-slave-q9j6 krb5kdc[15879](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781967064, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 14:51:04.696272 15985 init.cc:377] Logged in from keytab as kudu/127.0.97.1@KRBTEST.COM (short username kudu)
I20260620 14:51:04.697721 15985 webserver.cc:460] Webserver started at http://127.0.97.1:34223/ using document root <none> and password file <none>
I20260620 14:51:04.698378 15985 fs_manager.cc:359] Metadata directory not provided
I20260620 14:51:04.698480 15985 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 14:51:04.698762 15985 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 14:51:04.701785 15985 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskUci3DU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781966829091300-388-0/minicluster-data/ts-0/data/instance:
uuid: "531130dea5ed49f5926726011f1dc612"
format_stamp: "Formatted at 2026-06-20 14:51:04 on dist-test-slave-q9j6"
server_key: "5e2e2b609987b64726862da0486862ef"
server_key_iv: "8a2d2e54a4b49939c308f7beab05f356"
server_key_version: "encryptionkey@0"
I20260620 14:51:04.702508 15985 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskUci3DU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781966829091300-388-0/minicluster-data/ts-0/wal/instance:
uuid: "531130dea5ed49f5926726011f1dc612"
format_stamp: "Formatted at 2026-06-20 14:51:04 on dist-test-slave-q9j6"
server_key: "5e2e2b609987b64726862da0486862ef"
server_key_iv: "8a2d2e54a4b49939c308f7beab05f356"
server_key_version: "encryptionkey@0"
I20260620 14:51:04.707803 15985 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.002s sys 0.006s
I20260620 14:51:04.711218 16000 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 14:51:04.712718 15985 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260620 14:51:04.712855 15985 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskUci3DU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781966829091300-388-0/minicluster-data/ts-0/data,/tmp/dist-test-taskUci3DU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781966829091300-388-0/minicluster-data/ts-0/wal
uuid: "531130dea5ed49f5926726011f1dc612"
format_stamp: "Formatted at 2026-06-20 14:51:04 on dist-test-slave-q9j6"
server_key: "5e2e2b609987b64726862da0486862ef"
server_key_iv: "8a2d2e54a4b49939c308f7beab05f356"
server_key_version: "encryptionkey@0"
I20260620 14:51:04.713047 15985 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskUci3DU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781966829091300-388-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskUci3DU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781966829091300-388-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskUci3DU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781966829091300-388-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 14:51:04.729522 15985 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 14:51:04.734336 15985 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 14:51:04.734609 15985 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 14:51:04.735572 15985 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 14:51:04.736963 15985 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 14:51:04.737037 15985 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 14:51:04.737124 15985 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 14:51:04.737169 15985 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 14:51:04.776953 15985 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.1:40009
I20260620 14:51:04.774376 16112 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.1:40009 every 8 connection(s)
I20260620 14:51:04.779208 15985 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskUci3DU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781966829091300-388-0/minicluster-data/ts-0/data/info.pb
I20260620 14:51:04.788031 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskUci3DU/build/asan/bin/kudu as pid 15985
I20260620 14:51:04.788271 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskUci3DU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781966829091300-388-0/minicluster-data/ts-0/wal/instance
I20260620 14:51:04.789119 388 external_mini_cluster.cc:1428] Setting key 7404014ab3ad9c6d0cac078a624248c5
Jun 20 14:51:04 dist-test-slave-q9j6 krb5kdc[15879](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781967064, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.1@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260620 14:51:04.804112 16115 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:51:04.783128 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.1:58485 (local address 127.0.97.62:45085)
0620 14:51:04.783701 (+ 573us) server_negotiation.cc:206] Beginning negotiation
0620 14:51:04.783710 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0620 14:51:04.784356 (+ 646us) server_negotiation.cc:407] Connection header received
0620 14:51:04.786856 (+ 2500us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:51:04.786862 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:51:04.786959 (+ 97us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:51:04.787092 (+ 133us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:51:04.789446 (+ 2354us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:51:04.790395 (+ 949us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:51:04.792699 (+ 2304us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:51:04.793146 (+ 447us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:51:04.793250 (+ 104us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:51:04.797838 (+ 4588us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 14:51:04.797843 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 14:51:04.797846 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 14:51:04.797883 (+ 37us) server_negotiation.cc:922] calling sasl_server_start()
0620 14:51:04.800652 (+ 2769us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:51:04.801572 (+ 920us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:51:04.801576 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:51:04.801579 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:51:04.801713 (+ 134us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:51:04.802203 (+ 490us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:51:04.802208 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:51:04.802210 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:51:04.802493 (+ 283us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 14:51:04.802669 (+ 176us) server_negotiation.cc:1035] Waiting for connection context
0620 14:51:04.803455 (+ 786us) server_negotiation.cc:299] Negotiation successful
0620 14:51:04.803738 (+ 283us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":387,"thread_start_us":255,"threads_started":1}
I20260620 14:51:04.805224 16116 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:51:04.783202 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:45085 (local address 127.0.97.1:58485)
0620 14:51:04.784163 (+ 961us) negotiation.cc:107] Waiting for socket to connect
0620 14:51:04.784215 (+ 52us) client_negotiation.cc:174] Beginning negotiation
0620 14:51:04.786552 (+ 2337us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 14:51:04.787398 (+ 846us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 14:51:04.787407 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 14:51:04.788267 (+ 860us) client_negotiation.cc:189] Negotiated authn=SASL
0620 14:51:04.789226 (+ 959us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:51:04.789259 (+ 33us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:51:04.790589 (+ 1330us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:51:04.790593 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:51:04.792490 (+ 1897us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:51:04.792511 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:51:04.793406 (+ 895us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:51:04.793417 (+ 11us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:51:04.793778 (+ 361us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:51:04.794557 (+ 779us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 14:51:04.794593 (+ 36us) client_negotiation.cc:656] calling sasl_client_start()
0620 14:51:04.797559 (+ 2966us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 14:51:04.800871 (+ 3312us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 14:51:04.800884 (+ 13us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 14:51:04.800906 (+ 22us) client_negotiation.cc:760] calling sasl_client_step()
0620 14:51:04.801404 (+ 498us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 14:51:04.801838 (+ 434us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 14:51:04.801843 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 14:51:04.801847 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0620 14:51:04.802065 (+ 218us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 14:51:04.802639 (+ 574us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 14:51:04.802648 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 14:51:04.803054 (+ 406us) client_negotiation.cc:769] Sending connection context
0620 14:51:04.803348 (+ 294us) client_negotiation.cc:240] Negotiation successful
0620 14:51:04.803951 (+ 603us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":653,"thread_start_us":405,"threads_started":1}
I20260620 14:51:04.807113 16113 heartbeater.cc:347] Connected to a master server at 127.0.97.62:45085
I20260620 14:51:04.807508 16113 heartbeater.cc:464] Registering TS with master...
I20260620 14:51:04.808589 16113 heartbeater.cc:511] Master 127.0.97.62:45085 requested a full tablet report, sending...
I20260620 14:51:04.811194 15924 ts_manager.cc:194] Registered new tserver with Master: 531130dea5ed49f5926726011f1dc612 (127.0.97.1:40009)
I20260620 14:51:04.812815 15924 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.97.1@KRBTEST.COM'} at 127.0.97.1:58485
WARNING: no policy specified for kudu/127.0.97.2@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskUci3DU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781966829091300-388-0/krb5kdc/kudu_127.0.97.2.keytab.
Entry for principal kudu/127.0.97.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskUci3DU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781966829091300-388-0/krb5kdc/kudu_127.0.97.2.keytab.
I20260620 14:51:04.844398 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskUci3DU/build/asan/bin/kudu
/tmp/dist-test-taskUci3DU/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskUci3DU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781966829091300-388-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskUci3DU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781966829091300-388-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskUci3DU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781966829091300-388-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskUci3DU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781966829091300-388-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.2:0
--local_ip_for_outbound_sockets=127.0.97.2
--webserver_interface=127.0.97.2
--webserver_port=0
--tserver_master_addrs=127.0.97.62:45085
--keytab_file=/tmp/dist-test-taskUci3DU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781966829091300-388-0/krb5kdc/kudu_127.0.97.2.keytab
--principal=kudu/127.0.97.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:44619
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskUci3DU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781966829091300-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskUci3DU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781966829091300-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskUci3DU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781966829091300-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 14:51:05.182367 16119 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 14:51:05.182667 16119 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 14:51:05.182832 16119 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 14:51:05.190737 16119 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 14:51:05.190862 16119 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 14:51:05.191056 16119 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.2
I20260620 14:51:05.201079 16119 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:44619
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskUci3DU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781966829091300-388-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskUci3DU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781966829091300-388-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskUci3DU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781966829091300-388-0/krb5kdc/kudu_127.0.97.2.keytab
--principal=kudu/127.0.97.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskUci3DU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781966829091300-388-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.97.2
--webserver_port=0
--tserver_master_addrs=127.0.97.62:45085
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.2
--log_dir=/tmp/dist-test-taskUci3DU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781966829091300-388-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 13:41:20 UTC on ccycloud.quasar-vdagoi.root.comops.site
build id 4457462
ASAN enabled
I20260620 14:51:05.202394 16119 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 14:51:05.203784 16119 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 14:51:05.217226 16124 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 14:51:05.217343 16125 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 14:51:05.217378 16127 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 14:51:05.218389 16119 server_base.cc:1034] running on GCE node
I20260620 14:51:05.219039 16119 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 14:51:05.220118 16119 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 14:51:05.221378 16119 hybrid_clock.cc:648] HybridClock initialized: now 1781967065221332 us; error 43 us; skew 500 ppm
I20260620 14:51:05.221742 16119 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 14:51:05 dist-test-slave-q9j6 krb5kdc[15879](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781967065, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 14:51:05.225339 16119 init.cc:377] Logged in from keytab as kudu/127.0.97.2@KRBTEST.COM (short username kudu)
I20260620 14:51:05.226833 16119 webserver.cc:460] Webserver started at http://127.0.97.2:40609/ using document root <none> and password file <none>
I20260620 14:51:05.227703 16119 fs_manager.cc:359] Metadata directory not provided
I20260620 14:51:05.227819 16119 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 14:51:05.228113 16119 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 14:51:05.231163 16119 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskUci3DU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781966829091300-388-0/minicluster-data/ts-1/data/instance:
uuid: "ff16403da3304c18aff565fade86842e"
format_stamp: "Formatted at 2026-06-20 14:51:05 on dist-test-slave-q9j6"
server_key: "028d91d47f18dd8924ae2d375a9553de"
server_key_iv: "f326979f277e705d2dc12dfd1218c78c"
server_key_version: "encryptionkey@0"
I20260620 14:51:05.232059 16119 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskUci3DU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781966829091300-388-0/minicluster-data/ts-1/wal/instance:
uuid: "ff16403da3304c18aff565fade86842e"
format_stamp: "Formatted at 2026-06-20 14:51:05 on dist-test-slave-q9j6"
server_key: "028d91d47f18dd8924ae2d375a9553de"
server_key_iv: "f326979f277e705d2dc12dfd1218c78c"
server_key_version: "encryptionkey@0"
I20260620 14:51:05.237744 16119 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260620 14:51:05.241741 16134 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 14:51:05.243259 16119 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260620 14:51:05.243449 16119 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskUci3DU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781966829091300-388-0/minicluster-data/ts-1/data,/tmp/dist-test-taskUci3DU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781966829091300-388-0/minicluster-data/ts-1/wal
uuid: "ff16403da3304c18aff565fade86842e"
format_stamp: "Formatted at 2026-06-20 14:51:05 on dist-test-slave-q9j6"
server_key: "028d91d47f18dd8924ae2d375a9553de"
server_key_iv: "f326979f277e705d2dc12dfd1218c78c"
server_key_version: "encryptionkey@0"
I20260620 14:51:05.243736 16119 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskUci3DU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781966829091300-388-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskUci3DU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781966829091300-388-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskUci3DU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781966829091300-388-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 14:51:05.267290 16119 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 14:51:05.271986 16119 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 14:51:05.272302 16119 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 14:51:05.273689 16119 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 14:51:05.275242 16119 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 14:51:05.275327 16119 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 14:51:05.275480 16119 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 14:51:05.275542 16119 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 14:51:05.310040 16119 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.2:32783
I20260620 14:51:05.310092 16246 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.2:32783 every 8 connection(s)
I20260620 14:51:05.311390 16119 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskUci3DU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781966829091300-388-0/minicluster-data/ts-1/data/info.pb
I20260620 14:51:05.319528 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskUci3DU/build/asan/bin/kudu as pid 16119
I20260620 14:51:05.319770 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskUci3DU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781966829091300-388-0/minicluster-data/ts-1/wal/instance
I20260620 14:51:05.320369 388 external_mini_cluster.cc:1428] Setting key 28a7bbfe5532f7a30e84071d70bf79f4
Jun 20 14:51:05 dist-test-slave-q9j6 krb5kdc[15879](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781967065, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.2@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260620 14:51:05.334329 16249 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:51:05.314166 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.2:56283 (local address 127.0.97.62:45085)
0620 14:51:05.314639 (+ 473us) server_negotiation.cc:206] Beginning negotiation
0620 14:51:05.314643 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 14:51:05.315751 (+ 1108us) server_negotiation.cc:407] Connection header received
0620 14:51:05.317867 (+ 2116us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:51:05.317875 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:51:05.317975 (+ 100us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:51:05.318104 (+ 129us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:51:05.320170 (+ 2066us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:51:05.321049 (+ 879us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:51:05.323214 (+ 2165us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:51:05.323718 (+ 504us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:51:05.323819 (+ 101us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:51:05.328295 (+ 4476us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 14:51:05.328300 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 14:51:05.328303 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 14:51:05.328337 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0620 14:51:05.330985 (+ 2648us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:51:05.331870 (+ 885us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:51:05.331874 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:51:05.331877 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:51:05.332005 (+ 128us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:51:05.332477 (+ 472us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:51:05.332489 (+ 12us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:51:05.332491 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:51:05.332768 (+ 277us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 14:51:05.332933 (+ 165us) server_negotiation.cc:1035] Waiting for connection context
0620 14:51:05.333686 (+ 753us) server_negotiation.cc:299] Negotiation successful
0620 14:51:05.333929 (+ 243us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":358,"thread_start_us":250,"threads_started":1}
I20260620 14:51:05.335428 16250 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:51:05.314757 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:45085 (local address 127.0.97.2:56283)
0620 14:51:05.315512 (+ 755us) negotiation.cc:107] Waiting for socket to connect
0620 14:51:05.315555 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0620 14:51:05.317522 (+ 1967us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 14:51:05.318382 (+ 860us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 14:51:05.318391 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 14:51:05.318970 (+ 579us) client_negotiation.cc:189] Negotiated authn=SASL
0620 14:51:05.319921 (+ 951us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:51:05.319972 (+ 51us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:51:05.321267 (+ 1295us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:51:05.321272 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:51:05.323049 (+ 1777us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:51:05.323068 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:51:05.323976 (+ 908us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:51:05.323986 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:51:05.324371 (+ 385us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:51:05.325165 (+ 794us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 14:51:05.325202 (+ 37us) client_negotiation.cc:656] calling sasl_client_start()
0620 14:51:05.328024 (+ 2822us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 14:51:05.331171 (+ 3147us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 14:51:05.331180 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 14:51:05.331201 (+ 21us) client_negotiation.cc:760] calling sasl_client_step()
0620 14:51:05.331730 (+ 529us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 14:51:05.332138 (+ 408us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 14:51:05.332143 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 14:51:05.332147 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0620 14:51:05.332353 (+ 206us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 14:51:05.332907 (+ 554us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 14:51:05.332916 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 14:51:05.333295 (+ 379us) client_negotiation.cc:769] Sending connection context
0620 14:51:05.333594 (+ 299us) client_negotiation.cc:240] Negotiation successful
0620 14:51:05.334168 (+ 574us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":499,"thread_start_us":287,"threads_started":1}
I20260620 14:51:05.337081 16247 heartbeater.cc:347] Connected to a master server at 127.0.97.62:45085
I20260620 14:51:05.337409 16247 heartbeater.cc:464] Registering TS with master...
I20260620 14:51:05.338147 16247 heartbeater.cc:511] Master 127.0.97.62:45085 requested a full tablet report, sending...
I20260620 14:51:05.339839 15924 ts_manager.cc:194] Registered new tserver with Master: ff16403da3304c18aff565fade86842e (127.0.97.2:32783)
I20260620 14:51:05.340688 15924 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.97.2@KRBTEST.COM'} at 127.0.97.2:56283
WARNING: no policy specified for kudu/127.0.97.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskUci3DU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781966829091300-388-0/krb5kdc/kudu_127.0.97.3.keytab.
Entry for principal kudu/127.0.97.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskUci3DU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781966829091300-388-0/krb5kdc/kudu_127.0.97.3.keytab.
I20260620 14:51:05.374625 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskUci3DU/build/asan/bin/kudu
/tmp/dist-test-taskUci3DU/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskUci3DU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781966829091300-388-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskUci3DU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781966829091300-388-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskUci3DU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781966829091300-388-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskUci3DU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781966829091300-388-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.3:0
--local_ip_for_outbound_sockets=127.0.97.3
--webserver_interface=127.0.97.3
--webserver_port=0
--tserver_master_addrs=127.0.97.62:45085
--keytab_file=/tmp/dist-test-taskUci3DU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781966829091300-388-0/krb5kdc/kudu_127.0.97.3.keytab
--principal=kudu/127.0.97.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:44619
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskUci3DU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781966829091300-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskUci3DU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781966829091300-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskUci3DU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781966829091300-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 14:51:05.721774 16253 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 14:51:05.722091 16253 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 14:51:05.722187 16253 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 14:51:05.729758 16253 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 14:51:05.729889 16253 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 14:51:05.730013 16253 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.3
I20260620 14:51:05.740398 16253 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:44619
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskUci3DU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781966829091300-388-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskUci3DU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781966829091300-388-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskUci3DU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781966829091300-388-0/krb5kdc/kudu_127.0.97.3.keytab
--principal=kudu/127.0.97.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskUci3DU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781966829091300-388-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.97.3
--webserver_port=0
--tserver_master_addrs=127.0.97.62:45085
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.3
--log_dir=/tmp/dist-test-taskUci3DU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781966829091300-388-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 13:41:20 UTC on ccycloud.quasar-vdagoi.root.comops.site
build id 4457462
ASAN enabled
I20260620 14:51:05.741712 16253 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 14:51:05.743019 16253 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 14:51:05.756388 16258 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 14:51:05.756407 16261 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 14:51:05.756445 16259 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 14:51:05.757601 16253 server_base.cc:1034] running on GCE node
I20260620 14:51:05.758314 16253 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 14:51:05.759317 16253 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 14:51:05.760553 16253 hybrid_clock.cc:648] HybridClock initialized: now 1781967065760474 us; error 79 us; skew 500 ppm
I20260620 14:51:05.760893 16253 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 14:51:05 dist-test-slave-q9j6 krb5kdc[15879](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781967065, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 14:51:05.764353 16253 init.cc:377] Logged in from keytab as kudu/127.0.97.3@KRBTEST.COM (short username kudu)
I20260620 14:51:05.766039 16253 webserver.cc:460] Webserver started at http://127.0.97.3:36959/ using document root <none> and password file <none>
I20260620 14:51:05.766772 16253 fs_manager.cc:359] Metadata directory not provided
I20260620 14:51:05.766901 16253 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 14:51:05.767226 16253 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 14:51:05.770278 16253 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskUci3DU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781966829091300-388-0/minicluster-data/ts-2/data/instance:
uuid: "6879ba8da72645c29edc506ff924ba31"
format_stamp: "Formatted at 2026-06-20 14:51:05 on dist-test-slave-q9j6"
server_key: "d36a4e4bd2f0abafa50e8a532e9e4c45"
server_key_iv: "bac9ca16940dc3b8aa0a4af457f0ce41"
server_key_version: "encryptionkey@0"
I20260620 14:51:05.771076 16253 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskUci3DU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781966829091300-388-0/minicluster-data/ts-2/wal/instance:
uuid: "6879ba8da72645c29edc506ff924ba31"
format_stamp: "Formatted at 2026-06-20 14:51:05 on dist-test-slave-q9j6"
server_key: "d36a4e4bd2f0abafa50e8a532e9e4c45"
server_key_iv: "bac9ca16940dc3b8aa0a4af457f0ce41"
server_key_version: "encryptionkey@0"
I20260620 14:51:05.776974 16253 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.006s sys 0.002s
I20260620 14:51:05.780503 16268 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 14:51:05.781862 16253 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.001s
I20260620 14:51:05.782051 16253 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskUci3DU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781966829091300-388-0/minicluster-data/ts-2/data,/tmp/dist-test-taskUci3DU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781966829091300-388-0/minicluster-data/ts-2/wal
uuid: "6879ba8da72645c29edc506ff924ba31"
format_stamp: "Formatted at 2026-06-20 14:51:05 on dist-test-slave-q9j6"
server_key: "d36a4e4bd2f0abafa50e8a532e9e4c45"
server_key_iv: "bac9ca16940dc3b8aa0a4af457f0ce41"
server_key_version: "encryptionkey@0"
I20260620 14:51:05.782264 16253 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskUci3DU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781966829091300-388-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskUci3DU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781966829091300-388-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskUci3DU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781966829091300-388-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 14:51:05.801060 16253 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 14:51:05.805107 16253 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 14:51:05.805409 16253 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 14:51:05.806383 16253 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 14:51:05.807823 16253 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 14:51:05.807924 16253 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 14:51:05.808079 16253 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 14:51:05.808151 16253 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 14:51:05.816382 16113 heartbeater.cc:503] Master 127.0.97.62:45085 was elected leader, sending a full tablet report...
I20260620 14:51:05.843369 16253 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.3:33429
I20260620 14:51:05.843406 16380 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.3:33429 every 8 connection(s)
I20260620 14:51:05.844857 16253 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskUci3DU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781966829091300-388-0/minicluster-data/ts-2/data/info.pb
I20260620 14:51:05.849331 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskUci3DU/build/asan/bin/kudu as pid 16253
I20260620 14:51:05.849531 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskUci3DU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781966829091300-388-0/minicluster-data/ts-2/wal/instance
I20260620 14:51:05.850095 388 external_mini_cluster.cc:1428] Setting key f9406461f8da81858f24a07904b4666f
Jun 20 14:51:05 dist-test-slave-q9j6 krb5kdc[15879](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781967065, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.3@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260620 14:51:05.867015 16383 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:51:05.847661 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.3:42101 (local address 127.0.97.62:45085)
0620 14:51:05.848142 (+ 481us) server_negotiation.cc:206] Beginning negotiation
0620 14:51:05.848147 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 14:51:05.849293 (+ 1146us) server_negotiation.cc:407] Connection header received
0620 14:51:05.851699 (+ 2406us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:51:05.851705 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:51:05.851811 (+ 106us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:51:05.851928 (+ 117us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:51:05.853907 (+ 1979us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:51:05.854740 (+ 833us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:51:05.856388 (+ 1648us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:51:05.856799 (+ 411us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:51:05.856938 (+ 139us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:51:05.860729 (+ 3791us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 14:51:05.860734 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 14:51:05.860738 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 14:51:05.860774 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0620 14:51:05.863514 (+ 2740us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:51:05.864406 (+ 892us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:51:05.864410 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:51:05.864413 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:51:05.864540 (+ 127us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:51:05.865040 (+ 500us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:51:05.865045 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:51:05.865048 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:51:05.865390 (+ 342us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 14:51:05.865555 (+ 165us) server_negotiation.cc:1035] Waiting for connection context
0620 14:51:05.866378 (+ 823us) server_negotiation.cc:299] Negotiation successful
0620 14:51:05.866617 (+ 239us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":373,"thread_start_us":265,"threads_started":1}
I20260620 14:51:05.868077 16384 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:51:05.848234 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:45085 (local address 127.0.97.3:42101)
0620 14:51:05.849071 (+ 837us) negotiation.cc:107] Waiting for socket to connect
0620 14:51:05.849131 (+ 60us) client_negotiation.cc:174] Beginning negotiation
0620 14:51:05.851356 (+ 2225us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 14:51:05.852191 (+ 835us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 14:51:05.852200 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 14:51:05.852978 (+ 778us) client_negotiation.cc:189] Negotiated authn=SASL
0620 14:51:05.853741 (+ 763us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:51:05.853760 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:51:05.854901 (+ 1141us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:51:05.854908 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:51:05.856239 (+ 1331us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:51:05.856252 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:51:05.857039 (+ 787us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:51:05.857056 (+ 17us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:51:05.857287 (+ 231us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:51:05.858004 (+ 717us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 14:51:05.858027 (+ 23us) client_negotiation.cc:656] calling sasl_client_start()
0620 14:51:05.860474 (+ 2447us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 14:51:05.863725 (+ 3251us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 14:51:05.863732 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 14:51:05.863748 (+ 16us) client_negotiation.cc:760] calling sasl_client_step()
0620 14:51:05.864259 (+ 511us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 14:51:05.864720 (+ 461us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 14:51:05.864725 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 14:51:05.864729 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0620 14:51:05.864934 (+ 205us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 14:51:05.865538 (+ 604us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 14:51:05.865558 (+ 20us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 14:51:05.865961 (+ 403us) client_negotiation.cc:769] Sending connection context
0620 14:51:05.866277 (+ 316us) client_negotiation.cc:240] Negotiation successful
0620 14:51:05.866876 (+ 599us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":524,"thread_start_us":292,"threads_started":1}
I20260620 14:51:05.869400 16381 heartbeater.cc:347] Connected to a master server at 127.0.97.62:45085
I20260620 14:51:05.869735 16381 heartbeater.cc:464] Registering TS with master...
I20260620 14:51:05.870462 16381 heartbeater.cc:511] Master 127.0.97.62:45085 requested a full tablet report, sending...
I20260620 14:51:05.871893 15924 ts_manager.cc:194] Registered new tserver with Master: 6879ba8da72645c29edc506ff924ba31 (127.0.97.3:33429)
I20260620 14:51:05.872737 15924 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.97.3@KRBTEST.COM'} at 127.0.97.3:42101
I20260620 14:51:05.878444 388 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260620 14:51:05.896332 16383 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:51:05.883071 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60218 (local address 127.0.97.62:45085)
0620 14:51:05.883255 (+ 184us) server_negotiation.cc:206] Beginning negotiation
0620 14:51:05.883259 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 14:51:05.883754 (+ 495us) server_negotiation.cc:407] Connection header received
0620 14:51:05.883906 (+ 152us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:51:05.883910 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:51:05.883994 (+ 84us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:51:05.884086 (+ 92us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:51:05.885657 (+ 1571us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:51:05.886368 (+ 711us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:51:05.888072 (+ 1704us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:51:05.888598 (+ 526us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:51:05.888700 (+ 102us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:51:05.890949 (+ 2249us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 14:51:05.890955 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 14:51:05.890959 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 14:51:05.890994 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0620 14:51:05.893715 (+ 2721us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:51:05.894339 (+ 624us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:51:05.894342 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:51:05.894345 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:51:05.894458 (+ 113us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:51:05.894762 (+ 304us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:51:05.894765 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:51:05.894767 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:51:05.895076 (+ 309us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 14:51:05.895343 (+ 267us) server_negotiation.cc:1035] Waiting for connection context
0620 14:51:05.895767 (+ 424us) server_negotiation.cc:299] Negotiation successful
0620 14:51:05.896012 (+ 245us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":67}
I20260620 14:51:05.900503 15924 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:60218:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260620 14:51:05.907555 15924 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260620 14:51:05.931263 16394 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:51:05.920547 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:40009 (local address 127.0.0.1:48098)
0620 14:51:05.921411 (+ 864us) negotiation.cc:107] Waiting for socket to connect
0620 14:51:05.921466 (+ 55us) client_negotiation.cc:174] Beginning negotiation
0620 14:51:05.921701 (+ 235us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 14:51:05.923346 (+ 1645us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 14:51:05.923359 (+ 13us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 14:51:05.923401 (+ 42us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 14:51:05.923862 (+ 461us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:51:05.923881 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:51:05.925880 (+ 1999us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:51:05.925885 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:51:05.928250 (+ 2365us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:51:05.928266 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:51:05.930241 (+ 1975us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:51:05.930248 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:51:05.930445 (+ 197us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:51:05.930503 (+ 58us) client_negotiation.cc:769] Sending connection context
0620 14:51:05.930613 (+ 110us) client_negotiation.cc:240] Negotiation successful
0620 14:51:05.930809 (+ 196us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":538,"thread_start_us":347,"threads_started":1}
I20260620 14:51:05.931784 16397 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:51:05.922853 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.2:32783 (local address 127.0.0.1:52768)
0620 14:51:05.923404 (+ 551us) negotiation.cc:107] Waiting for socket to connect
0620 14:51:05.923422 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0620 14:51:05.923531 (+ 109us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 14:51:05.924728 (+ 1197us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 14:51:05.924733 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 14:51:05.924757 (+ 24us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 14:51:05.925126 (+ 369us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:51:05.925138 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:51:05.927109 (+ 1971us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:51:05.927114 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:51:05.929186 (+ 2072us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:51:05.929208 (+ 22us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:51:05.930844 (+ 1636us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:51:05.930850 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:51:05.931042 (+ 192us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:51:05.931080 (+ 38us) client_negotiation.cc:769] Sending connection context
0620 14:51:05.931185 (+ 105us) client_negotiation.cc:240] Negotiation successful
0620 14:51:05.931327 (+ 142us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":420,"mutex_wait_us":27,"thread_start_us":526,"threads_started":1}
I20260620 14:51:05.932309 16395 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:51:05.920638 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48098 (local address 127.0.97.1:40009)
0620 14:51:05.922308 (+ 1670us) server_negotiation.cc:206] Beginning negotiation
0620 14:51:05.922317 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0620 14:51:05.922346 (+ 29us) server_negotiation.cc:407] Connection header received
0620 14:51:05.922446 (+ 100us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:51:05.922454 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:51:05.922738 (+ 284us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:51:05.922930 (+ 192us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 14:51:05.924070 (+ 1140us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:51:05.925734 (+ 1664us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:51:05.928438 (+ 2704us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:51:05.930016 (+ 1578us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:51:05.931460 (+ 1444us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:51:05.931715 (+ 255us) server_negotiation.cc:1035] Waiting for connection context
0620 14:51:05.931793 (+ 78us) server_negotiation.cc:299] Negotiation successful
0620 14:51:05.931983 (+ 190us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1499,"thread_start_us":1714,"threads_started":1}
I20260620 14:51:05.932406 16398 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:51:05.922753 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43598 (local address 127.0.97.3:33429)
0620 14:51:05.924535 (+ 1782us) server_negotiation.cc:206] Beginning negotiation
0620 14:51:05.924547 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0620 14:51:05.924571 (+ 24us) server_negotiation.cc:407] Connection header received
0620 14:51:05.924641 (+ 70us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:51:05.924646 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:51:05.924794 (+ 148us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:51:05.924979 (+ 185us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 14:51:05.925702 (+ 723us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:51:05.927218 (+ 1516us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:51:05.929564 (+ 2346us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:51:05.930939 (+ 1375us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:51:05.931103 (+ 164us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:51:05.931332 (+ 229us) server_negotiation.cc:1035] Waiting for connection context
0620 14:51:05.931895 (+ 563us) server_negotiation.cc:299] Negotiation successful
0620 14:51:05.932084 (+ 189us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1619,"thread_start_us":1460,"threads_started":1}
I20260620 14:51:05.932468 16399 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:51:05.923404 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52768 (local address 127.0.97.2:32783)
0620 14:51:05.923886 (+ 482us) server_negotiation.cc:206] Beginning negotiation
0620 14:51:05.923892 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 14:51:05.923913 (+ 21us) server_negotiation.cc:407] Connection header received
0620 14:51:05.924024 (+ 111us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:51:05.924031 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:51:05.924285 (+ 254us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:51:05.924512 (+ 227us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 14:51:05.925357 (+ 845us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:51:05.926951 (+ 1594us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:51:05.929408 (+ 2457us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:51:05.930625 (+ 1217us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:51:05.930783 (+ 158us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:51:05.931000 (+ 217us) server_negotiation.cc:1035] Waiting for connection context
0620 14:51:05.932018 (+ 1018us) server_negotiation.cc:299] Negotiation successful
0620 14:51:05.932176 (+ 158us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":363,"thread_start_us":276,"threads_started":1}
I20260620 14:51:05.932801 16396 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:51:05.922113 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.3:33429 (local address 127.0.0.1:43598)
0620 14:51:05.922682 (+ 569us) negotiation.cc:107] Waiting for socket to connect
0620 14:51:05.922701 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0620 14:51:05.922806 (+ 105us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 14:51:05.924970 (+ 2164us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 14:51:05.924975 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 14:51:05.924999 (+ 24us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 14:51:05.925495 (+ 496us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:51:05.925506 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:51:05.927356 (+ 1850us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:51:05.927361 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:51:05.929352 (+ 1991us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:51:05.929368 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:51:05.931183 (+ 1815us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:51:05.931189 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:51:05.931352 (+ 163us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:51:05.931380 (+ 28us) client_negotiation.cc:769] Sending connection context
0620 14:51:05.931470 (+ 90us) client_negotiation.cc:240] Negotiation successful
0620 14:51:05.931618 (+ 148us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":443,"mutex_wait_us":4,"thread_start_us":1110,"threads_started":1}
I20260620 14:51:05.936390 16316 tablet_service.cc:1467] Processing CreateTablet for tablet f0b72f41b6e0405ba643578796791fa1 (DEFAULT_TABLE table=test-table [id=3cff1ff4144a49c7adbe83aa6459f11e]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 14:51:05.936403 16182 tablet_service.cc:1467] Processing CreateTablet for tablet f0b72f41b6e0405ba643578796791fa1 (DEFAULT_TABLE table=test-table [id=3cff1ff4144a49c7adbe83aa6459f11e]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 14:51:05.936403 16048 tablet_service.cc:1467] Processing CreateTablet for tablet f0b72f41b6e0405ba643578796791fa1 (DEFAULT_TABLE table=test-table [id=3cff1ff4144a49c7adbe83aa6459f11e]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 14:51:05.937749 16316 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f0b72f41b6e0405ba643578796791fa1. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 14:51:05.937762 16048 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f0b72f41b6e0405ba643578796791fa1. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 14:51:05.937780 16182 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f0b72f41b6e0405ba643578796791fa1. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 14:51:05.949945 16400 tablet_bootstrap.cc:492] T f0b72f41b6e0405ba643578796791fa1 P 6879ba8da72645c29edc506ff924ba31: Bootstrap starting.
I20260620 14:51:05.954279 16400 tablet_bootstrap.cc:654] T f0b72f41b6e0405ba643578796791fa1 P 6879ba8da72645c29edc506ff924ba31: Neither blocks nor log segments found. Creating new log.
I20260620 14:51:05.955406 16400 log.cc:824] T f0b72f41b6e0405ba643578796791fa1 P 6879ba8da72645c29edc506ff924ba31: Log is configured to *not* fsync() on all Append() calls
I20260620 14:51:05.956472 16402 tablet_bootstrap.cc:492] T f0b72f41b6e0405ba643578796791fa1 P 531130dea5ed49f5926726011f1dc612: Bootstrap starting.
I20260620 14:51:05.958688 16401 tablet_bootstrap.cc:492] T f0b72f41b6e0405ba643578796791fa1 P ff16403da3304c18aff565fade86842e: Bootstrap starting.
I20260620 14:51:05.960412 16400 tablet_bootstrap.cc:492] T f0b72f41b6e0405ba643578796791fa1 P 6879ba8da72645c29edc506ff924ba31: No bootstrap required, opened a new log
I20260620 14:51:05.960716 16400 ts_tablet_manager.cc:1397] T f0b72f41b6e0405ba643578796791fa1 P 6879ba8da72645c29edc506ff924ba31: Time spent bootstrapping tablet: real 0.011s user 0.009s sys 0.000s
I20260620 14:51:05.960834 16402 tablet_bootstrap.cc:654] T f0b72f41b6e0405ba643578796791fa1 P 531130dea5ed49f5926726011f1dc612: Neither blocks nor log segments found. Creating new log.
I20260620 14:51:05.962244 16402 log.cc:824] T f0b72f41b6e0405ba643578796791fa1 P 531130dea5ed49f5926726011f1dc612: Log is configured to *not* fsync() on all Append() calls
I20260620 14:51:05.962415 16401 tablet_bootstrap.cc:654] T f0b72f41b6e0405ba643578796791fa1 P ff16403da3304c18aff565fade86842e: Neither blocks nor log segments found. Creating new log.
I20260620 14:51:05.963832 16401 log.cc:824] T f0b72f41b6e0405ba643578796791fa1 P ff16403da3304c18aff565fade86842e: Log is configured to *not* fsync() on all Append() calls
I20260620 14:51:05.966012 16402 tablet_bootstrap.cc:492] T f0b72f41b6e0405ba643578796791fa1 P 531130dea5ed49f5926726011f1dc612: No bootstrap required, opened a new log
I20260620 14:51:05.966338 16402 ts_tablet_manager.cc:1397] T f0b72f41b6e0405ba643578796791fa1 P 531130dea5ed49f5926726011f1dc612: Time spent bootstrapping tablet: real 0.010s user 0.008s sys 0.000s
I20260620 14:51:05.967351 16401 tablet_bootstrap.cc:492] T f0b72f41b6e0405ba643578796791fa1 P ff16403da3304c18aff565fade86842e: No bootstrap required, opened a new log
I20260620 14:51:05.967665 16401 ts_tablet_manager.cc:1397] T f0b72f41b6e0405ba643578796791fa1 P ff16403da3304c18aff565fade86842e: Time spent bootstrapping tablet: real 0.009s user 0.009s sys 0.000s
I20260620 14:51:05.969831 16400 raft_consensus.cc:348] T f0b72f41b6e0405ba643578796791fa1 P 6879ba8da72645c29edc506ff924ba31 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "531130dea5ed49f5926726011f1dc612" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 40009 } } peers { permanent_uuid: "6879ba8da72645c29edc506ff924ba31" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 33429 } } peers { permanent_uuid: "ff16403da3304c18aff565fade86842e" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 32783 } }
I20260620 14:51:05.970250 16400 raft_consensus.cc:374] T f0b72f41b6e0405ba643578796791fa1 P 6879ba8da72645c29edc506ff924ba31 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 14:51:05.970391 16400 raft_consensus.cc:729] T f0b72f41b6e0405ba643578796791fa1 P 6879ba8da72645c29edc506ff924ba31 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6879ba8da72645c29edc506ff924ba31, State: Initialized, Role: FOLLOWER
I20260620 14:51:05.971060 16400 consensus_queue.cc:260] T f0b72f41b6e0405ba643578796791fa1 P 6879ba8da72645c29edc506ff924ba31 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "531130dea5ed49f5926726011f1dc612" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 40009 } } peers { permanent_uuid: "6879ba8da72645c29edc506ff924ba31" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 33429 } } peers { permanent_uuid: "ff16403da3304c18aff565fade86842e" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 32783 } }
I20260620 14:51:05.973035 16400 ts_tablet_manager.cc:1428] T f0b72f41b6e0405ba643578796791fa1 P 6879ba8da72645c29edc506ff924ba31: Time spent starting tablet: real 0.012s user 0.007s sys 0.006s
I20260620 14:51:05.973680 16381 heartbeater.cc:503] Master 127.0.97.62:45085 was elected leader, sending a full tablet report...
I20260620 14:51:05.978780 16401 raft_consensus.cc:348] T f0b72f41b6e0405ba643578796791fa1 P ff16403da3304c18aff565fade86842e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "531130dea5ed49f5926726011f1dc612" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 40009 } } peers { permanent_uuid: "6879ba8da72645c29edc506ff924ba31" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 33429 } } peers { permanent_uuid: "ff16403da3304c18aff565fade86842e" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 32783 } }
I20260620 14:51:05.978991 16402 raft_consensus.cc:348] T f0b72f41b6e0405ba643578796791fa1 P 531130dea5ed49f5926726011f1dc612 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "531130dea5ed49f5926726011f1dc612" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 40009 } } peers { permanent_uuid: "6879ba8da72645c29edc506ff924ba31" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 33429 } } peers { permanent_uuid: "ff16403da3304c18aff565fade86842e" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 32783 } }
I20260620 14:51:05.979348 16401 raft_consensus.cc:374] T f0b72f41b6e0405ba643578796791fa1 P ff16403da3304c18aff565fade86842e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 14:51:05.979386 16402 raft_consensus.cc:374] T f0b72f41b6e0405ba643578796791fa1 P 531130dea5ed49f5926726011f1dc612 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 14:51:05.979471 16402 raft_consensus.cc:729] T f0b72f41b6e0405ba643578796791fa1 P 531130dea5ed49f5926726011f1dc612 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 531130dea5ed49f5926726011f1dc612, State: Initialized, Role: FOLLOWER
I20260620 14:51:05.979471 16401 raft_consensus.cc:729] T f0b72f41b6e0405ba643578796791fa1 P ff16403da3304c18aff565fade86842e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ff16403da3304c18aff565fade86842e, State: Initialized, Role: FOLLOWER
I20260620 14:51:05.980194 16402 consensus_queue.cc:260] T f0b72f41b6e0405ba643578796791fa1 P 531130dea5ed49f5926726011f1dc612 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "531130dea5ed49f5926726011f1dc612" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 40009 } } peers { permanent_uuid: "6879ba8da72645c29edc506ff924ba31" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 33429 } } peers { permanent_uuid: "ff16403da3304c18aff565fade86842e" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 32783 } }
I20260620 14:51:05.980194 16401 consensus_queue.cc:260] T f0b72f41b6e0405ba643578796791fa1 P ff16403da3304c18aff565fade86842e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "531130dea5ed49f5926726011f1dc612" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 40009 } } peers { permanent_uuid: "6879ba8da72645c29edc506ff924ba31" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 33429 } } peers { permanent_uuid: "ff16403da3304c18aff565fade86842e" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 32783 } }
I20260620 14:51:05.981500 16247 heartbeater.cc:503] Master 127.0.97.62:45085 was elected leader, sending a full tablet report...
I20260620 14:51:05.982280 16402 ts_tablet_manager.cc:1428] T f0b72f41b6e0405ba643578796791fa1 P 531130dea5ed49f5926726011f1dc612: Time spent starting tablet: real 0.016s user 0.010s sys 0.005s
I20260620 14:51:05.982306 16401 ts_tablet_manager.cc:1428] T f0b72f41b6e0405ba643578796791fa1 P ff16403da3304c18aff565fade86842e: Time spent starting tablet: real 0.015s user 0.013s sys 0.000s
W20260620 14:51:06.033463 16114 tablet.cc:2369] T f0b72f41b6e0405ba643578796791fa1 P 531130dea5ed49f5926726011f1dc612: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260620 14:51:06.038625 16406 raft_consensus.cc:482] T f0b72f41b6e0405ba643578796791fa1 P 6879ba8da72645c29edc506ff924ba31 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260620 14:51:06.038947 16406 raft_consensus.cc:504] T f0b72f41b6e0405ba643578796791fa1 P 6879ba8da72645c29edc506ff924ba31 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "531130dea5ed49f5926726011f1dc612" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 40009 } } peers { permanent_uuid: "6879ba8da72645c29edc506ff924ba31" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 33429 } } peers { permanent_uuid: "ff16403da3304c18aff565fade86842e" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 32783 } }
I20260620 14:51:06.040828 16406 leader_election.cc:288] T f0b72f41b6e0405ba643578796791fa1 P 6879ba8da72645c29edc506ff924ba31 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 531130dea5ed49f5926726011f1dc612 (127.0.97.1:40009), ff16403da3304c18aff565fade86842e (127.0.97.2:32783)
I20260620 14:51:06.046635 16384 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:51:06.041211 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:40009 (local address 127.0.97.3:54673)
0620 14:51:06.041480 (+ 269us) negotiation.cc:107] Waiting for socket to connect
0620 14:51:06.041502 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0620 14:51:06.041701 (+ 199us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 14:51:06.042261 (+ 560us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 14:51:06.042273 (+ 12us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 14:51:06.042295 (+ 22us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 14:51:06.042453 (+ 158us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:51:06.042461 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:51:06.043427 (+ 966us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:51:06.043433 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:51:06.044766 (+ 1333us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:51:06.044780 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:51:06.045906 (+ 1126us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:51:06.045912 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:51:06.046099 (+ 187us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:51:06.046123 (+ 24us) client_negotiation.cc:769] Sending connection context
0620 14:51:06.046230 (+ 107us) client_negotiation.cc:240] Negotiation successful
0620 14:51:06.046364 (+ 134us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":86}
I20260620 14:51:06.046639 16395 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:51:06.041229 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.3:54673 (local address 127.0.97.1:40009)
0620 14:51:06.041468 (+ 239us) server_negotiation.cc:206] Beginning negotiation
0620 14:51:06.041476 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0620 14:51:06.041593 (+ 117us) server_negotiation.cc:407] Connection header received
0620 14:51:06.041890 (+ 297us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:51:06.041899 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:51:06.042006 (+ 107us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:51:06.042113 (+ 107us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 14:51:06.042597 (+ 484us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:51:06.043295 (+ 698us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:51:06.044946 (+ 1651us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:51:06.045736 (+ 790us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:51:06.045884 (+ 148us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:51:06.046011 (+ 127us) server_negotiation.cc:1035] Waiting for connection context
0620 14:51:06.046255 (+ 244us) server_negotiation.cc:299] Negotiation successful
0620 14:51:06.046382 (+ 127us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":67}
I20260620 14:51:06.048070 16409 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:51:06.041628 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.2:32783 (local address 127.0.97.3:53697)
0620 14:51:06.042367 (+ 739us) negotiation.cc:107] Waiting for socket to connect
0620 14:51:06.042381 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0620 14:51:06.042535 (+ 154us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 14:51:06.042897 (+ 362us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 14:51:06.042901 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 14:51:06.042924 (+ 23us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 14:51:06.043246 (+ 322us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:51:06.043254 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:51:06.044390 (+ 1136us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:51:06.044402 (+ 12us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:51:06.045722 (+ 1320us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:51:06.045734 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:51:06.047315 (+ 1581us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:51:06.047322 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:51:06.047495 (+ 173us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:51:06.047532 (+ 37us) client_negotiation.cc:769] Sending connection context
0620 14:51:06.047666 (+ 134us) client_negotiation.cc:240] Negotiation successful
0620 14:51:06.047805 (+ 139us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":627,"thread_start_us":414,"threads_started":1}
I20260620 14:51:06.048144 16399 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:51:06.041974 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.3:53697 (local address 127.0.97.2:32783)
0620 14:51:06.042202 (+ 228us) server_negotiation.cc:206] Beginning negotiation
0620 14:51:06.042209 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0620 14:51:06.042475 (+ 266us) server_negotiation.cc:407] Connection header received
0620 14:51:06.042668 (+ 193us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:51:06.042680 (+ 12us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:51:06.042777 (+ 97us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:51:06.042890 (+ 113us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 14:51:06.043393 (+ 503us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:51:06.044224 (+ 831us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:51:06.045890 (+ 1666us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:51:06.047030 (+ 1140us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:51:06.047198 (+ 168us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:51:06.047294 (+ 96us) server_negotiation.cc:1035] Waiting for connection context
0620 14:51:06.047717 (+ 423us) server_negotiation.cc:299] Negotiation successful
0620 14:51:06.047883 (+ 166us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":80}
I20260620 14:51:06.048102 16068 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f0b72f41b6e0405ba643578796791fa1" candidate_uuid: "6879ba8da72645c29edc506ff924ba31" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "531130dea5ed49f5926726011f1dc612" is_pre_election: true
I20260620 14:51:06.048764 16068 raft_consensus.cc:2454] T f0b72f41b6e0405ba643578796791fa1 P 531130dea5ed49f5926726011f1dc612 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6879ba8da72645c29edc506ff924ba31 in term 0.
I20260620 14:51:06.049247 16202 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f0b72f41b6e0405ba643578796791fa1" candidate_uuid: "6879ba8da72645c29edc506ff924ba31" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ff16403da3304c18aff565fade86842e" is_pre_election: true
I20260620 14:51:06.049547 16272 leader_election.cc:302] T f0b72f41b6e0405ba643578796791fa1 P 6879ba8da72645c29edc506ff924ba31 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 531130dea5ed49f5926726011f1dc612, 6879ba8da72645c29edc506ff924ba31; no voters:
I20260620 14:51:06.049741 16202 raft_consensus.cc:2454] T f0b72f41b6e0405ba643578796791fa1 P ff16403da3304c18aff565fade86842e [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6879ba8da72645c29edc506ff924ba31 in term 0.
I20260620 14:51:06.049950 16406 raft_consensus.cc:2781] T f0b72f41b6e0405ba643578796791fa1 P 6879ba8da72645c29edc506ff924ba31 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260620 14:51:06.050100 16406 raft_consensus.cc:482] T f0b72f41b6e0405ba643578796791fa1 P 6879ba8da72645c29edc506ff924ba31 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260620 14:51:06.050232 16406 raft_consensus.cc:3037] T f0b72f41b6e0405ba643578796791fa1 P 6879ba8da72645c29edc506ff924ba31 [term 0 FOLLOWER]: Advancing to term 1
I20260620 14:51:06.052606 16406 raft_consensus.cc:504] T f0b72f41b6e0405ba643578796791fa1 P 6879ba8da72645c29edc506ff924ba31 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "531130dea5ed49f5926726011f1dc612" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 40009 } } peers { permanent_uuid: "6879ba8da72645c29edc506ff924ba31" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 33429 } } peers { permanent_uuid: "ff16403da3304c18aff565fade86842e" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 32783 } }
I20260620 14:51:06.053218 16406 leader_election.cc:288] T f0b72f41b6e0405ba643578796791fa1 P 6879ba8da72645c29edc506ff924ba31 [CANDIDATE]: Term 1 election: Requested vote from peers 531130dea5ed49f5926726011f1dc612 (127.0.97.1:40009), ff16403da3304c18aff565fade86842e (127.0.97.2:32783)
I20260620 14:51:06.053797 16068 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f0b72f41b6e0405ba643578796791fa1" candidate_uuid: "6879ba8da72645c29edc506ff924ba31" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "531130dea5ed49f5926726011f1dc612"
I20260620 14:51:06.053876 16202 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f0b72f41b6e0405ba643578796791fa1" candidate_uuid: "6879ba8da72645c29edc506ff924ba31" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ff16403da3304c18aff565fade86842e"
I20260620 14:51:06.054018 16068 raft_consensus.cc:3037] T f0b72f41b6e0405ba643578796791fa1 P 531130dea5ed49f5926726011f1dc612 [term 0 FOLLOWER]: Advancing to term 1
I20260620 14:51:06.054136 16202 raft_consensus.cc:3037] T f0b72f41b6e0405ba643578796791fa1 P ff16403da3304c18aff565fade86842e [term 0 FOLLOWER]: Advancing to term 1
I20260620 14:51:06.056609 16202 raft_consensus.cc:2454] T f0b72f41b6e0405ba643578796791fa1 P ff16403da3304c18aff565fade86842e [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6879ba8da72645c29edc506ff924ba31 in term 1.
I20260620 14:51:06.056643 16068 raft_consensus.cc:2454] T f0b72f41b6e0405ba643578796791fa1 P 531130dea5ed49f5926726011f1dc612 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6879ba8da72645c29edc506ff924ba31 in term 1.
I20260620 14:51:06.057153 16272 leader_election.cc:302] T f0b72f41b6e0405ba643578796791fa1 P 6879ba8da72645c29edc506ff924ba31 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6879ba8da72645c29edc506ff924ba31, ff16403da3304c18aff565fade86842e; no voters:
I20260620 14:51:06.057510 16406 raft_consensus.cc:2781] T f0b72f41b6e0405ba643578796791fa1 P 6879ba8da72645c29edc506ff924ba31 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 14:51:06.057957 16406 raft_consensus.cc:686] T f0b72f41b6e0405ba643578796791fa1 P 6879ba8da72645c29edc506ff924ba31 [term 1 LEADER]: Becoming Leader. State: Replica: 6879ba8da72645c29edc506ff924ba31, State: Running, Role: LEADER
I20260620 14:51:06.058498 16406 consensus_queue.cc:237] T f0b72f41b6e0405ba643578796791fa1 P 6879ba8da72645c29edc506ff924ba31 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "531130dea5ed49f5926726011f1dc612" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 40009 } } peers { permanent_uuid: "6879ba8da72645c29edc506ff924ba31" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 33429 } } peers { permanent_uuid: "ff16403da3304c18aff565fade86842e" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 32783 } }
W20260620 14:51:06.063383 16248 tablet.cc:2369] T f0b72f41b6e0405ba643578796791fa1 P ff16403da3304c18aff565fade86842e: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260620 14:51:06.063252 15923 catalog_manager.cc:5495] T f0b72f41b6e0405ba643578796791fa1 P 6879ba8da72645c29edc506ff924ba31 reported cstate change: term changed from 0 to 1, leader changed from <none> to 6879ba8da72645c29edc506ff924ba31 (127.0.97.3). New cstate: current_term: 1 leader_uuid: "6879ba8da72645c29edc506ff924ba31" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "531130dea5ed49f5926726011f1dc612" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 40009 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "6879ba8da72645c29edc506ff924ba31" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 33429 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "ff16403da3304c18aff565fade86842e" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 32783 } health_report { overall_health: UNKNOWN } } }
I20260620 14:51:06.089958 16398 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:51:06.084461 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43614 (local address 127.0.97.3:33429)
0620 14:51:06.084684 (+ 223us) server_negotiation.cc:206] Beginning negotiation
0620 14:51:06.084689 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 14:51:06.084707 (+ 18us) server_negotiation.cc:407] Connection header received
0620 14:51:06.084890 (+ 183us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:51:06.084894 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:51:06.084972 (+ 78us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:51:06.085095 (+ 123us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 14:51:06.085482 (+ 387us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:51:06.086175 (+ 693us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:51:06.087752 (+ 1577us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:51:06.088170 (+ 418us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:51:06.088262 (+ 92us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:51:06.088758 (+ 496us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 14:51:06.089372 (+ 614us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0620 14:51:06.089483 (+ 111us) server_negotiation.cc:1035] Waiting for connection context
0620 14:51:06.089596 (+ 113us) server_negotiation.cc:299] Negotiation successful
0620 14:51:06.089662 (+ 66us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":87}
W20260620 14:51:06.096495 16382 tablet.cc:2369] T f0b72f41b6e0405ba643578796791fa1 P 6879ba8da72645c29edc506ff924ba31: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260620 14:51:06.098677 16202 raft_consensus.cc:1261] T f0b72f41b6e0405ba643578796791fa1 P ff16403da3304c18aff565fade86842e [term 1 FOLLOWER]: Refusing update from remote peer 6879ba8da72645c29edc506ff924ba31: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 14:51:06.098677 16068 raft_consensus.cc:1261] T f0b72f41b6e0405ba643578796791fa1 P 531130dea5ed49f5926726011f1dc612 [term 1 FOLLOWER]: Refusing update from remote peer 6879ba8da72645c29edc506ff924ba31: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 14:51:06.099808 16410 consensus_queue.cc:1035] T f0b72f41b6e0405ba643578796791fa1 P 6879ba8da72645c29edc506ff924ba31 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ff16403da3304c18aff565fade86842e" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 32783 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 14:51:06.100189 16406 consensus_queue.cc:1035] T f0b72f41b6e0405ba643578796791fa1 P 6879ba8da72645c29edc506ff924ba31 [LEADER]: Connected to new peer: Peer: permanent_uuid: "531130dea5ed49f5926726011f1dc612" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 40009 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 14:51:06.115775 16414 mvcc.cc:205] Tried to move back new op lower bound from 7298937102728679424 to 7298937102579380224. Current Snapshot: MvccSnapshot[applied={T|T < 7298937102728679424}]
I20260620 14:51:06.135198 15922 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:60218:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 14:51:06.135567 15922 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:60218:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 14:51:06.140241 15922 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 0069e3cceda94cd68d41c5833a5174c9: Sending DeleteTablet for 3 replicas of tablet f0b72f41b6e0405ba643578796791fa1
I20260620 14:51:06.141592 16182 tablet_service.cc:1514] Processing DeleteTablet for tablet f0b72f41b6e0405ba643578796791fa1 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 14:51:06 UTC) from {username='kudu', principal='kudu/127.0.97.62@KRBTEST.COM'} at 127.0.0.1:52768
I20260620 14:51:06.141574 16316 tablet_service.cc:1514] Processing DeleteTablet for tablet f0b72f41b6e0405ba643578796791fa1 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 14:51:06 UTC) from {username='kudu', principal='kudu/127.0.97.62@KRBTEST.COM'} at 127.0.0.1:43598
I20260620 14:51:06.141754 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskUci3DU/build/asan/bin/kudu pid 15985
I20260620 14:51:06.142428 16048 tablet_service.cc:1514] Processing DeleteTablet for tablet f0b72f41b6e0405ba643578796791fa1 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 14:51:06 UTC) from {username='kudu', principal='kudu/127.0.97.62@KRBTEST.COM'} at 127.0.0.1:48098
I20260620 14:51:06.142998 16422 tablet_replica.cc:330] T f0b72f41b6e0405ba643578796791fa1 P ff16403da3304c18aff565fade86842e: stopping tablet replica
I20260620 14:51:06.143713 16422 raft_consensus.cc:2229] T f0b72f41b6e0405ba643578796791fa1 P ff16403da3304c18aff565fade86842e [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 14:51:06.144531 16422 raft_consensus.cc:2258] T f0b72f41b6e0405ba643578796791fa1 P ff16403da3304c18aff565fade86842e [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 14:51:06.147555 16422 ts_tablet_manager.cc:1905] T f0b72f41b6e0405ba643578796791fa1 P ff16403da3304c18aff565fade86842e: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 14:51:06.148670 16423 tablet_replica.cc:330] T f0b72f41b6e0405ba643578796791fa1 P 6879ba8da72645c29edc506ff924ba31: stopping tablet replica
I20260620 14:51:06.152130 16423 raft_consensus.cc:2229] T f0b72f41b6e0405ba643578796791fa1 P 6879ba8da72645c29edc506ff924ba31 [term 1 LEADER]: Raft consensus shutting down.
I20260620 14:51:06.153113 16423 raft_consensus.cc:2258] T f0b72f41b6e0405ba643578796791fa1 P 6879ba8da72645c29edc506ff924ba31 [term 1 FOLLOWER]: Raft consensus is shut down!
Jun 20 14:51:06 dist-test-slave-q9j6 krb5kdc[15879](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781967063, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.1@KRBTEST.COM
I20260620 14:51:06.155545 16423 ts_tablet_manager.cc:1905] T f0b72f41b6e0405ba643578796791fa1 P 6879ba8da72645c29edc506ff924ba31: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 14:51:06.156776 16422 ts_tablet_manager.cc:1918] T f0b72f41b6e0405ba643578796791fa1 P ff16403da3304c18aff565fade86842e: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260620 14:51:06.156998 16422 log.cc:1196] T f0b72f41b6e0405ba643578796791fa1 P ff16403da3304c18aff565fade86842e: Deleting WAL directory at /tmp/dist-test-taskUci3DU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781966829091300-388-0/minicluster-data/ts-1/wal/wals/f0b72f41b6e0405ba643578796791fa1
I20260620 14:51:06.157558 16422 ts_tablet_manager.cc:1939] T f0b72f41b6e0405ba643578796791fa1 P ff16403da3304c18aff565fade86842e: Deleting consensus metadata
I20260620 14:51:06.159744 15911 catalog_manager.cc:4841] TS ff16403da3304c18aff565fade86842e (127.0.97.2:32783): tablet f0b72f41b6e0405ba643578796791fa1 (table test-table [id=3cff1ff4144a49c7adbe83aa6459f11e]) successfully deleted
I20260620 14:51:06.161594 16425 tablet_replica.cc:330] T f0b72f41b6e0405ba643578796791fa1 P 531130dea5ed49f5926726011f1dc612: stopping tablet replica
I20260620 14:51:06.162038 16425 raft_consensus.cc:2229] T f0b72f41b6e0405ba643578796791fa1 P 531130dea5ed49f5926726011f1dc612 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 14:51:06.162549 16425 raft_consensus.cc:2258] T f0b72f41b6e0405ba643578796791fa1 P 531130dea5ed49f5926726011f1dc612 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 14:51:06.164371 16395 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:51:06.142586 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48106 (local address 127.0.97.1:40009)
0620 14:51:06.142786 (+ 200us) server_negotiation.cc:206] Beginning negotiation
0620 14:51:06.142805 (+ 19us) server_negotiation.cc:399] Waiting for connection header
0620 14:51:06.143319 (+ 514us) server_negotiation.cc:407] Connection header received
0620 14:51:06.143684 (+ 365us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:51:06.143693 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:51:06.143800 (+ 107us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:51:06.143915 (+ 115us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:51:06.146578 (+ 2663us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:51:06.148159 (+ 1581us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:51:06.149739 (+ 1580us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:51:06.150169 (+ 430us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:51:06.150320 (+ 151us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:51:06.156160 (+ 5840us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 14:51:06.156182 (+ 22us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 14:51:06.156204 (+ 22us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 14:51:06.156264 (+ 60us) server_negotiation.cc:922] calling sasl_server_start()
0620 14:51:06.160335 (+ 4071us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:51:06.161430 (+ 1095us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:51:06.161440 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:51:06.161444 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:51:06.161644 (+ 200us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:51:06.162195 (+ 551us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:51:06.162201 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:51:06.162204 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:51:06.162794 (+ 590us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 14:51:06.163319 (+ 525us) server_negotiation.cc:1035] Waiting for connection context
0620 14:51:06.163681 (+ 362us) server_negotiation.cc:299] Negotiation successful
0620 14:51:06.163948 (+ 267us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":68}
I20260620 14:51:06.164283 16423 ts_tablet_manager.cc:1918] T f0b72f41b6e0405ba643578796791fa1 P 6879ba8da72645c29edc506ff924ba31: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260620 14:51:06.164520 16423 log.cc:1196] T f0b72f41b6e0405ba643578796791fa1 P 6879ba8da72645c29edc506ff924ba31: Deleting WAL directory at /tmp/dist-test-taskUci3DU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781966829091300-388-0/minicluster-data/ts-2/wal/wals/f0b72f41b6e0405ba643578796791fa1
I20260620 14:51:06.164964 16425 ts_tablet_manager.cc:1905] T f0b72f41b6e0405ba643578796791fa1 P 531130dea5ed49f5926726011f1dc612: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 14:51:06.165077 16423 ts_tablet_manager.cc:1939] T f0b72f41b6e0405ba643578796791fa1 P 6879ba8da72645c29edc506ff924ba31: Deleting consensus metadata
I20260620 14:51:06.165115 16108 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 14:51:06.167466 15909 catalog_manager.cc:4841] TS 6879ba8da72645c29edc506ff924ba31 (127.0.97.3:33429): tablet f0b72f41b6e0405ba643578796791fa1 (table test-table [id=3cff1ff4144a49c7adbe83aa6459f11e]) successfully deleted
I20260620 14:51:06.256212 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskUci3DU/build/asan/bin/kudu with pid 15985
I20260620 14:51:06.274889 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskUci3DU/build/asan/bin/kudu pid 16119
W20260620 14:51:06.274893 15911 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.0.97.1:40009 (error 108)
I20260620 14:51:06.277376 16396 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:51:06.276864 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:40009 (local address 127.0.0.1:48120)
0620 14:51:06.277043 (+ 179us) negotiation.cc:107] Waiting for socket to connect
0620 14:51:06.277167 (+ 124us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.1:40009: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":64}
W20260620 14:51:06.277629 15911 catalog_manager.cc:4584] TS 531130dea5ed49f5926726011f1dc612 (127.0.97.1:40009): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet f0b72f41b6e0405ba643578796791fa1: Network error: Client connection negotiation failed: client connection to 127.0.97.1:40009: connect: Connection refused (error 111)
Jun 20 14:51:06 dist-test-slave-q9j6 krb5kdc[15879](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781967063, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.2@KRBTEST.COM
I20260620 14:51:06.294924 16399 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:51:06.276348 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52770 (local address 127.0.97.2:32783)
0620 14:51:06.276536 (+ 188us) server_negotiation.cc:206] Beginning negotiation
0620 14:51:06.276542 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 14:51:06.276562 (+ 20us) server_negotiation.cc:407] Connection header received
0620 14:51:06.276769 (+ 207us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:51:06.276775 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:51:06.276863 (+ 88us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:51:06.276970 (+ 107us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:51:06.279018 (+ 2048us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:51:06.280303 (+ 1285us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:51:06.281923 (+ 1620us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:51:06.282592 (+ 669us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:51:06.282736 (+ 144us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:51:06.287443 (+ 4707us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 14:51:06.287466 (+ 23us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 14:51:06.287487 (+ 21us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 14:51:06.287547 (+ 60us) server_negotiation.cc:922] calling sasl_server_start()
0620 14:51:06.291585 (+ 4038us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:51:06.292563 (+ 978us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:51:06.292575 (+ 12us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:51:06.292578 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:51:06.292768 (+ 190us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:51:06.293155 (+ 387us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:51:06.293160 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:51:06.293164 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:51:06.293747 (+ 583us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 14:51:06.294083 (+ 336us) server_negotiation.cc:1035] Waiting for connection context
0620 14:51:06.294293 (+ 210us) server_negotiation.cc:299] Negotiation successful
0620 14:51:06.294535 (+ 242us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":67}
I20260620 14:51:06.295732 16242 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 14:51:06.331449 16396 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:51:06.330831 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:40009 (local address 127.0.0.1:48136)
0620 14:51:06.331121 (+ 290us) negotiation.cc:107] Waiting for socket to connect
0620 14:51:06.331251 (+ 130us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.1:40009: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":105}
I20260620 14:51:06.332736 16396 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:51:06.332310 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:40009 (local address 127.0.0.1:48146)
0620 14:51:06.332484 (+ 174us) negotiation.cc:107] Waiting for socket to connect
0620 14:51:06.332588 (+ 104us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.1:40009: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":61}
I20260620 14:51:06.387413 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskUci3DU/build/asan/bin/kudu with pid 16119
I20260620 14:51:06.393435 16396 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:51:06.392938 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:40009 (local address 127.0.0.1:48160)
0620 14:51:06.393178 (+ 240us) negotiation.cc:107] Waiting for socket to connect
0620 14:51:06.393262 (+ 84us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.1:40009: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":77}
I20260620 14:51:06.394557 16396 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:51:06.394198 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:40009 (local address 127.0.0.1:48176)
0620 14:51:06.394354 (+ 156us) negotiation.cc:107] Waiting for socket to connect
0620 14:51:06.394447 (+ 93us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.1:40009: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":55}
I20260620 14:51:06.410921 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskUci3DU/build/asan/bin/kudu pid 16253
Jun 20 14:51:06 dist-test-slave-q9j6 krb5kdc[15879](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781967063, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.3@KRBTEST.COM
I20260620 14:51:06.430756 16398 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:51:06.411776 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43624 (local address 127.0.97.3:33429)
0620 14:51:06.411970 (+ 194us) server_negotiation.cc:206] Beginning negotiation
0620 14:51:06.411974 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 14:51:06.412049 (+ 75us) server_negotiation.cc:407] Connection header received
0620 14:51:06.412249 (+ 200us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:51:06.412253 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:51:06.412319 (+ 66us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:51:06.412429 (+ 110us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:51:06.414359 (+ 1930us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:51:06.415239 (+ 880us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:51:06.417451 (+ 2212us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:51:06.417922 (+ 471us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:51:06.418046 (+ 124us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:51:06.424105 (+ 6059us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 14:51:06.424118 (+ 13us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 14:51:06.424135 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 14:51:06.424182 (+ 47us) server_negotiation.cc:922] calling sasl_server_start()
0620 14:51:06.427529 (+ 3347us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:51:06.428471 (+ 942us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:51:06.428479 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:51:06.428481 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:51:06.428665 (+ 184us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:51:06.429086 (+ 421us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:51:06.429090 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:51:06.429092 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:51:06.429486 (+ 394us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 14:51:06.429825 (+ 339us) server_negotiation.cc:1035] Waiting for connection context
0620 14:51:06.430115 (+ 290us) server_negotiation.cc:299] Negotiation successful
0620 14:51:06.430353 (+ 238us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":66}
I20260620 14:51:06.431660 16376 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 14:51:06.475844 16396 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:51:06.475312 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:40009 (local address 127.0.0.1:48192)
0620 14:51:06.475529 (+ 217us) negotiation.cc:107] Waiting for socket to connect
0620 14:51:06.475686 (+ 157us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.1:40009: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":83}
I20260620 14:51:06.476913 16396 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:51:06.476491 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:40009 (local address 127.0.0.1:48196)
0620 14:51:06.476725 (+ 234us) negotiation.cc:107] Waiting for socket to connect
0620 14:51:06.476798 (+ 73us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.1:40009: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":104}
I20260620 14:51:06.554524 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskUci3DU/build/asan/bin/kudu with pid 16253
I20260620 14:51:06.571133 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskUci3DU/build/asan/bin/kudu pid 15892
I20260620 14:51:06.572395 15954 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 14:51:06.691354 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskUci3DU/build/asan/bin/kudu with pid 15892
I20260620 14:51:06.691390 16396 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:51:06.690663 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:40009 (local address 127.0.0.1:48200)
0620 14:51:06.691020 (+ 357us) negotiation.cc:107] Waiting for socket to connect
0620 14:51:06.691147 (+ 127us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.1:40009: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":203}
I20260620 14:51:26.713766 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskUci3DU/build/asan/bin/kudu
/tmp/dist-test-taskUci3DU/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskUci3DU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781966829091300-388-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskUci3DU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781966829091300-388-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskUci3DU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781966829091300-388-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskUci3DU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781966829091300-388-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:45085
--webserver_interface=127.0.97.62
--webserver_port=32859
--keytab_file=/tmp/dist-test-taskUci3DU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781966829091300-388-0/krb5kdc/kudu_127.0.97.62.keytab
--principal=kudu/127.0.97.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:44619
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.97.62:45085
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_rotation_seconds=20
--tsk_private_key_password_cmd=echo foo with env {KRB5CCNAME=/tmp/dist-test-taskUci3DU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781966829091300-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskUci3DU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781966829091300-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskUci3DU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781966829091300-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 14:51:27.075291 16431 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 14:51:27.075723 16431 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 14:51:27.075845 16431 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 14:51:27.083590 16431 flags.cc:425] Enabled experimental flag: --authn_token_validity_seconds=20
W20260620 14:51:27.083791 16431 flags.cc:425] Enabled experimental flag: --authz_token_validity_seconds=20
W20260620 14:51:27.083832 16431 flags.cc:425] Enabled experimental flag: --tsk_rotation_seconds=20
W20260620 14:51:27.083877 16431 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 14:51:27.083933 16431 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 14:51:27.084003 16431 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 14:51:27.084048 16431 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 14:51:27.084267 16431 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 14:51:27.095006 16431 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:44619
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskUci3DU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781966829091300-388-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskUci3DU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781966829091300-388-0/minicluster-data/master-0/wal
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_private_key_password_cmd=echo foo
--tsk_rotation_seconds=20
--ipki_ca_key_size=768
--master_addresses=127.0.97.62:45085
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskUci3DU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781966829091300-388-0/krb5kdc/kudu_127.0.97.62.keytab
--principal=kudu/127.0.97.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:45085
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskUci3DU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781966829091300-388-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.97.62
--webserver_port=32859
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskUci3DU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781966829091300-388-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 13:41:20 UTC on ccycloud.quasar-vdagoi.root.comops.site
build id 4457462
ASAN enabled
I20260620 14:51:27.096446 16431 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 14:51:27.097787 16431 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 14:51:27.121860 16440 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 14:51:27.121825 16438 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 14:51:27.121882 16437 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 14:51:27.125315 16431 server_base.cc:1034] running on GCE node
I20260620 14:51:27.126322 16431 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 14:51:27.128136 16431 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 14:51:27.129405 16431 hybrid_clock.cc:648] HybridClock initialized: now 1781967087129337 us; error 53 us; skew 500 ppm
I20260620 14:51:27.129827 16431 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 14:51:27 dist-test-slave-q9j6 krb5kdc[15879](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781967087, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 14:51:27.133533 16431 init.cc:377] Logged in from keytab as kudu/127.0.97.62@KRBTEST.COM (short username kudu)
I20260620 14:51:27.135380 16431 webserver.cc:460] Webserver started at http://127.0.97.62:32859/ using document root <none> and password file <none>
I20260620 14:51:27.136348 16431 fs_manager.cc:359] Metadata directory not provided
I20260620 14:51:27.136511 16431 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 14:51:27.142959 16431 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.005s sys 0.001s
I20260620 14:51:27.146279 16447 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 14:51:27.147761 16431 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260620 14:51:27.147969 16431 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskUci3DU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781966829091300-388-0/minicluster-data/master-0/data,/tmp/dist-test-taskUci3DU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781966829091300-388-0/minicluster-data/master-0/wal
uuid: "0069e3cceda94cd68d41c5833a5174c9"
format_stamp: "Formatted at 2026-06-20 14:51:04 on dist-test-slave-q9j6"
server_key: "8a231203b28462a6eed146cdec780c4f"
server_key_iv: "1bb0c6765e03213cde26645f51bf2628"
server_key_version: "encryptionkey@0"
I20260620 14:51:27.149070 16431 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskUci3DU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781966829091300-388-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskUci3DU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781966829091300-388-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskUci3DU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781966829091300-388-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 14:51:27.175010 16431 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 14:51:27.180076 16431 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 14:51:27.180399 16431 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 14:51:27.200881 16431 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.62:45085
I20260620 14:51:27.200899 16498 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.62:45085 every 8 connection(s)
I20260620 14:51:27.202652 16431 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskUci3DU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781966829091300-388-0/minicluster-data/master-0/data/info.pb
I20260620 14:51:27.209424 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskUci3DU/build/asan/bin/kudu as pid 16431
I20260620 14:51:27.210467 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskUci3DU/build/asan/bin/kudu
/tmp/dist-test-taskUci3DU/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskUci3DU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781966829091300-388-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskUci3DU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781966829091300-388-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskUci3DU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781966829091300-388-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskUci3DU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781966829091300-388-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.1:40009
--local_ip_for_outbound_sockets=127.0.97.1
--tserver_master_addrs=127.0.97.62:45085
--webserver_port=34223
--webserver_interface=127.0.97.1
--keytab_file=/tmp/dist-test-taskUci3DU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781966829091300-388-0/krb5kdc/kudu_127.0.97.1.keytab
--principal=kudu/127.0.97.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:44619
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskUci3DU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781966829091300-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskUci3DU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781966829091300-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskUci3DU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781966829091300-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260620 14:51:27.211095 16499 sys_catalog.cc:263] Verifying existing consensus state
I20260620 14:51:27.216789 16499 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 0069e3cceda94cd68d41c5833a5174c9: Bootstrap starting.
I20260620 14:51:27.238297 16499 log.cc:824] T 00000000000000000000000000000000 P 0069e3cceda94cd68d41c5833a5174c9: Log is configured to *not* fsync() on all Append() calls
I20260620 14:51:27.251485 16499 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 0069e3cceda94cd68d41c5833a5174c9: Bootstrap replayed 1/1 log segments. Stats: ops{read=8 overwritten=0 applied=8 ignored=0} inserts{seen=5 ignored=0} mutations{seen=4 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260620 14:51:27.252246 16499 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 0069e3cceda94cd68d41c5833a5174c9: Bootstrap complete.
I20260620 14:51:27.262104 16499 raft_consensus.cc:348] T 00000000000000000000000000000000 P 0069e3cceda94cd68d41c5833a5174c9 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0069e3cceda94cd68d41c5833a5174c9" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 45085 } }
I20260620 14:51:27.263031 16499 raft_consensus.cc:729] T 00000000000000000000000000000000 P 0069e3cceda94cd68d41c5833a5174c9 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0069e3cceda94cd68d41c5833a5174c9, State: Initialized, Role: FOLLOWER
I20260620 14:51:27.263825 16499 consensus_queue.cc:260] T 00000000000000000000000000000000 P 0069e3cceda94cd68d41c5833a5174c9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 8, Last appended: 1.8, Last appended by leader: 8, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0069e3cceda94cd68d41c5833a5174c9" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 45085 } }
I20260620 14:51:27.264088 16499 raft_consensus.cc:388] T 00000000000000000000000000000000 P 0069e3cceda94cd68d41c5833a5174c9 [term 1 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 14:51:27.264209 16499 raft_consensus.cc:482] T 00000000000000000000000000000000 P 0069e3cceda94cd68d41c5833a5174c9 [term 1 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 14:51:27.264407 16499 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 0069e3cceda94cd68d41c5833a5174c9 [term 1 FOLLOWER]: Advancing to term 2
I20260620 14:51:27.268965 16499 raft_consensus.cc:504] T 00000000000000000000000000000000 P 0069e3cceda94cd68d41c5833a5174c9 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0069e3cceda94cd68d41c5833a5174c9" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 45085 } }
I20260620 14:51:27.269553 16499 leader_election.cc:302] T 00000000000000000000000000000000 P 0069e3cceda94cd68d41c5833a5174c9 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 0069e3cceda94cd68d41c5833a5174c9; no voters:
I20260620 14:51:27.270308 16499 leader_election.cc:288] T 00000000000000000000000000000000 P 0069e3cceda94cd68d41c5833a5174c9 [CANDIDATE]: Term 2 election: Requested vote from peers
I20260620 14:51:27.270448 16503 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 0069e3cceda94cd68d41c5833a5174c9 [term 2 FOLLOWER]: Leader election won for term 2
I20260620 14:51:27.271745 16503 raft_consensus.cc:686] T 00000000000000000000000000000000 P 0069e3cceda94cd68d41c5833a5174c9 [term 2 LEADER]: Becoming Leader. State: Replica: 0069e3cceda94cd68d41c5833a5174c9, State: Running, Role: LEADER
I20260620 14:51:27.272308 16503 consensus_queue.cc:237] T 00000000000000000000000000000000 P 0069e3cceda94cd68d41c5833a5174c9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 1.8, Last appended by leader: 8, Current term: 2, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0069e3cceda94cd68d41c5833a5174c9" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 45085 } }
I20260620 14:51:27.272838 16499 sys_catalog.cc:564] T 00000000000000000000000000000000 P 0069e3cceda94cd68d41c5833a5174c9 [sys.catalog]: configured and running, proceeding with master startup.
I20260620 14:51:27.274844 16505 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0069e3cceda94cd68d41c5833a5174c9 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 0069e3cceda94cd68d41c5833a5174c9. Latest consensus state: current_term: 2 leader_uuid: "0069e3cceda94cd68d41c5833a5174c9" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0069e3cceda94cd68d41c5833a5174c9" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 45085 } } }
I20260620 14:51:27.275218 16505 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0069e3cceda94cd68d41c5833a5174c9 [sys.catalog]: This master's current role is: LEADER
I20260620 14:51:27.274976 16504 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0069e3cceda94cd68d41c5833a5174c9 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 2 leader_uuid: "0069e3cceda94cd68d41c5833a5174c9" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0069e3cceda94cd68d41c5833a5174c9" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 45085 } } }
I20260620 14:51:27.275454 16504 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0069e3cceda94cd68d41c5833a5174c9 [sys.catalog]: This master's current role is: LEADER
I20260620 14:51:27.288998 16508 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 14:51:27.297333 16508 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 14:51:27.299201 16508 catalog_manager.cc:1259] Loaded cluster ID: 85e8db040d0d4fe1baaac9f6b8446522
I20260620 14:51:27.299391 16508 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 14:51:27.303480 16508 catalog_manager.cc:1504] Loading token signing keys...
I20260620 14:51:27.305406 16508 catalog_manager.cc:5879] T 00000000000000000000000000000000 P 0069e3cceda94cd68d41c5833a5174c9: Loaded TSK: 0
I20260620 14:51:27.339001 16508 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 0069e3cceda94cd68d41c5833a5174c9: Generated new TSK 1
I20260620 14:51:27.339450 16508 catalog_manager.cc:1514] Initializing in-progress tserver states...
W20260620 14:51:27.590329 16501 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 14:51:27.590690 16501 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 14:51:27.590866 16501 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 14:51:27.598781 16501 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 14:51:27.598965 16501 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 14:51:27.599144 16501 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.1
I20260620 14:51:27.609853 16501 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:44619
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskUci3DU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781966829091300-388-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskUci3DU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781966829091300-388-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskUci3DU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781966829091300-388-0/krb5kdc/kudu_127.0.97.1.keytab
--principal=kudu/127.0.97.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.1:40009
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskUci3DU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781966829091300-388-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.97.1
--webserver_port=34223
--tserver_master_addrs=127.0.97.62:45085
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.1
--log_dir=/tmp/dist-test-taskUci3DU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781966829091300-388-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 13:41:20 UTC on ccycloud.quasar-vdagoi.root.comops.site
build id 4457462
ASAN enabled
I20260620 14:51:27.611277 16501 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 14:51:27.612830 16501 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 14:51:27.628551 16527 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 14:51:27.628582 16526 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 14:51:27.629326 16529 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 14:51:27.629885 16501 server_base.cc:1034] running on GCE node
I20260620 14:51:27.630555 16501 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 14:51:27.631759 16501 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 14:51:27.632999 16501 hybrid_clock.cc:648] HybridClock initialized: now 1781967087632928 us; error 72 us; skew 500 ppm
I20260620 14:51:27.633360 16501 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 14:51:27 dist-test-slave-q9j6 krb5kdc[15879](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781967087, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 14:51:27.637086 16501 init.cc:377] Logged in from keytab as kudu/127.0.97.1@KRBTEST.COM (short username kudu)
I20260620 14:51:27.639043 16501 webserver.cc:460] Webserver started at http://127.0.97.1:34223/ using document root <none> and password file <none>
I20260620 14:51:27.640036 16501 fs_manager.cc:359] Metadata directory not provided
I20260620 14:51:27.640204 16501 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 14:51:27.647698 16501 fs_manager.cc:709] Time spent opening directory manager: real 0.005s user 0.003s sys 0.005s
I20260620 14:51:27.651487 16536 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 14:51:27.653388 16501 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.000s sys 0.003s
I20260620 14:51:27.653600 16501 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskUci3DU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781966829091300-388-0/minicluster-data/ts-0/data,/tmp/dist-test-taskUci3DU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781966829091300-388-0/minicluster-data/ts-0/wal
uuid: "531130dea5ed49f5926726011f1dc612"
format_stamp: "Formatted at 2026-06-20 14:51:04 on dist-test-slave-q9j6"
server_key: "5e2e2b609987b64726862da0486862ef"
server_key_iv: "8a2d2e54a4b49939c308f7beab05f356"
server_key_version: "encryptionkey@0"
I20260620 14:51:27.654773 16501 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskUci3DU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781966829091300-388-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskUci3DU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781966829091300-388-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskUci3DU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781966829091300-388-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 14:51:27.674630 16501 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 14:51:27.679247 16501 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 14:51:27.679564 16501 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 14:51:27.680847 16501 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 14:51:27.682927 16543 ts_tablet_manager.cc:542] Loading tablet metadata (0/1 complete)
I20260620 14:51:27.689255 16501 ts_tablet_manager.cc:579] Loaded tablet metadata (1 total tablets, 1 live tablets)
I20260620 14:51:27.689388 16501 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.007s user 0.001s sys 0.000s
I20260620 14:51:27.689486 16501 ts_tablet_manager.cc:594] Registering tablets (0/1 complete)
I20260620 14:51:27.692873 16501 ts_tablet_manager.cc:610] Registered 1 tablets
I20260620 14:51:27.692987 16501 ts_tablet_manager.cc:589] Time spent register tablets: real 0.004s user 0.002s sys 0.000s
I20260620 14:51:27.693331 16543 tablet_bootstrap.cc:492] T f0b72f41b6e0405ba643578796791fa1 P 531130dea5ed49f5926726011f1dc612: Bootstrap starting.
I20260620 14:51:27.704824 16543 log.cc:824] T f0b72f41b6e0405ba643578796791fa1 P 531130dea5ed49f5926726011f1dc612: Log is configured to *not* fsync() on all Append() calls
I20260620 14:51:27.725077 16543 tablet_bootstrap.cc:492] T f0b72f41b6e0405ba643578796791fa1 P 531130dea5ed49f5926726011f1dc612: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=1 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260620 14:51:27.726069 16543 tablet_bootstrap.cc:492] T f0b72f41b6e0405ba643578796791fa1 P 531130dea5ed49f5926726011f1dc612: Bootstrap complete.
I20260620 14:51:27.726764 16543 ts_tablet_manager.cc:1397] T f0b72f41b6e0405ba643578796791fa1 P 531130dea5ed49f5926726011f1dc612: Time spent bootstrapping tablet: real 0.034s user 0.022s sys 0.008s
I20260620 14:51:27.735426 16501 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.1:40009
I20260620 14:51:27.735448 16650 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.1:40009 every 8 connection(s)
I20260620 14:51:27.737344 16501 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskUci3DU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781966829091300-388-0/minicluster-data/ts-0/data/info.pb
I20260620 14:51:27.739867 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskUci3DU/build/asan/bin/kudu as pid 16501
I20260620 14:51:27.740661 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskUci3DU/build/asan/bin/kudu
/tmp/dist-test-taskUci3DU/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskUci3DU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781966829091300-388-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskUci3DU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781966829091300-388-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskUci3DU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781966829091300-388-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskUci3DU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781966829091300-388-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.2:32783
--local_ip_for_outbound_sockets=127.0.97.2
--tserver_master_addrs=127.0.97.62:45085
--webserver_port=40609
--webserver_interface=127.0.97.2
--keytab_file=/tmp/dist-test-taskUci3DU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781966829091300-388-0/krb5kdc/kudu_127.0.97.2.keytab
--principal=kudu/127.0.97.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:44619
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskUci3DU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781966829091300-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskUci3DU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781966829091300-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskUci3DU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781966829091300-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260620 14:51:27.739952 16543 raft_consensus.cc:348] T f0b72f41b6e0405ba643578796791fa1 P 531130dea5ed49f5926726011f1dc612 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "531130dea5ed49f5926726011f1dc612" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 40009 } } peers { permanent_uuid: "6879ba8da72645c29edc506ff924ba31" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 33429 } } peers { permanent_uuid: "ff16403da3304c18aff565fade86842e" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 32783 } }
I20260620 14:51:27.741798 16543 raft_consensus.cc:729] T f0b72f41b6e0405ba643578796791fa1 P 531130dea5ed49f5926726011f1dc612 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 531130dea5ed49f5926726011f1dc612, State: Initialized, Role: FOLLOWER
I20260620 14:51:27.742985 16543 consensus_queue.cc:260] T f0b72f41b6e0405ba643578796791fa1 P 531130dea5ed49f5926726011f1dc612 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "531130dea5ed49f5926726011f1dc612" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 40009 } } peers { permanent_uuid: "6879ba8da72645c29edc506ff924ba31" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 33429 } } peers { permanent_uuid: "ff16403da3304c18aff565fade86842e" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 32783 } }
I20260620 14:51:27.748935 16543 ts_tablet_manager.cc:1428] T f0b72f41b6e0405ba643578796791fa1 P 531130dea5ed49f5926726011f1dc612: Time spent starting tablet: real 0.022s user 0.018s sys 0.005s
Jun 20 14:51:27 dist-test-slave-q9j6 krb5kdc[15879](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781967087, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.1@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260620 14:51:27.772562 16653 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:51:27.740860 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.1:53437 (local address 127.0.97.62:45085)
0620 14:51:27.752080 (+ 11220us) server_negotiation.cc:206] Beginning negotiation
0620 14:51:27.752095 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0620 14:51:27.752138 (+ 43us) server_negotiation.cc:407] Connection header received
0620 14:51:27.752472 (+ 334us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:51:27.752489 (+ 17us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:51:27.753014 (+ 525us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:51:27.753533 (+ 519us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:51:27.755479 (+ 1946us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:51:27.756861 (+ 1382us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:51:27.758822 (+ 1961us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:51:27.759637 (+ 815us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:51:27.759872 (+ 235us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:51:27.764120 (+ 4248us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 14:51:27.764129 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 14:51:27.764148 (+ 19us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 14:51:27.764195 (+ 47us) server_negotiation.cc:922] calling sasl_server_start()
0620 14:51:27.767268 (+ 3073us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:51:27.768609 (+ 1341us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:51:27.768617 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:51:27.768623 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:51:27.768813 (+ 190us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:51:27.769394 (+ 581us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:51:27.769397 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:51:27.769399 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:51:27.769920 (+ 521us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 14:51:27.770063 (+ 143us) server_negotiation.cc:1035] Waiting for connection context
0620 14:51:27.771108 (+ 1045us) server_negotiation.cc:299] Negotiation successful
0620 14:51:27.771464 (+ 356us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":10989,"thread_start_us":10890,"threads_started":1}
I20260620 14:51:27.772898 16654 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:51:27.740749 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:45085 (local address 127.0.97.1:53437)
0620 14:51:27.750099 (+ 9350us) negotiation.cc:107] Waiting for socket to connect
0620 14:51:27.750160 (+ 61us) client_negotiation.cc:174] Beginning negotiation
0620 14:51:27.750758 (+ 598us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 14:51:27.753689 (+ 2931us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 14:51:27.753704 (+ 15us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 14:51:27.754548 (+ 844us) client_negotiation.cc:189] Negotiated authn=SASL
0620 14:51:27.755208 (+ 660us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:51:27.755230 (+ 22us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:51:27.757065 (+ 1835us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:51:27.757072 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:51:27.758643 (+ 1571us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:51:27.758658 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:51:27.759859 (+ 1201us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:51:27.759866 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:51:27.760181 (+ 315us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:51:27.760935 (+ 754us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 14:51:27.760959 (+ 24us) client_negotiation.cc:656] calling sasl_client_start()
0620 14:51:27.763749 (+ 2790us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 14:51:27.767507 (+ 3758us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 14:51:27.767521 (+ 14us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 14:51:27.767545 (+ 24us) client_negotiation.cc:760] calling sasl_client_step()
0620 14:51:27.768439 (+ 894us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 14:51:27.768963 (+ 524us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 14:51:27.768972 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 14:51:27.768976 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0620 14:51:27.769261 (+ 285us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 14:51:27.770085 (+ 824us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 14:51:27.770097 (+ 12us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 14:51:27.770584 (+ 487us) client_negotiation.cc:769] Sending connection context
0620 14:51:27.770915 (+ 331us) client_negotiation.cc:240] Negotiation successful
0620 14:51:27.771482 (+ 567us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":9000,"thread_start_us":8737,"threads_started":1}
I20260620 14:51:27.776386 16651 heartbeater.cc:347] Connected to a master server at 127.0.97.62:45085
I20260620 14:51:27.776736 16651 heartbeater.cc:464] Registering TS with master...
I20260620 14:51:27.777519 16651 heartbeater.cc:511] Master 127.0.97.62:45085 requested a full tablet report, sending...
I20260620 14:51:27.781303 16464 ts_manager.cc:194] Registered new tserver with Master: 531130dea5ed49f5926726011f1dc612 (127.0.97.1:40009)
I20260620 14:51:27.787815 16464 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.97.1@KRBTEST.COM'} at 127.0.97.1:53437
I20260620 14:51:27.791707 16651 heartbeater.cc:503] Master 127.0.97.62:45085 was elected leader, sending a full tablet report...
Jun 20 14:51:27 dist-test-slave-q9j6 krb5kdc[15879](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781967087, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.62@KRBTEST.COM for kudu/127.0.97.1@KRBTEST.COM
I20260620 14:51:27.804831 16657 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:51:27.786041 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39910 (local address 127.0.97.1:40009)
0620 14:51:27.787075 (+ 1034us) server_negotiation.cc:206] Beginning negotiation
0620 14:51:27.787088 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0620 14:51:27.787743 (+ 655us) server_negotiation.cc:407] Connection header received
0620 14:51:27.788201 (+ 458us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:51:27.788210 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:51:27.788424 (+ 214us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:51:27.788527 (+ 103us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:51:27.790117 (+ 1590us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:51:27.791881 (+ 1764us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:51:27.793638 (+ 1757us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:51:27.794368 (+ 730us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:51:27.794506 (+ 138us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:51:27.798503 (+ 3997us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 14:51:27.798518 (+ 15us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 14:51:27.798531 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 14:51:27.798571 (+ 40us) server_negotiation.cc:922] calling sasl_server_start()
0620 14:51:27.801467 (+ 2896us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:51:27.802549 (+ 1082us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:51:27.802556 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:51:27.802558 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:51:27.802684 (+ 126us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:51:27.803108 (+ 424us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:51:27.803111 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:51:27.803113 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:51:27.803521 (+ 408us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 14:51:27.803750 (+ 229us) server_negotiation.cc:1035] Waiting for connection context
0620 14:51:27.804292 (+ 542us) server_negotiation.cc:299] Negotiation successful
0620 14:51:27.804527 (+ 235us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":793,"thread_start_us":309,"threads_started":1}
I20260620 14:51:27.804925 16658 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:51:27.786004 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:40009 (local address 127.0.0.1:39910)
0620 14:51:27.787546 (+ 1542us) negotiation.cc:107] Waiting for socket to connect
0620 14:51:27.787592 (+ 46us) client_negotiation.cc:174] Beginning negotiation
0620 14:51:27.787915 (+ 323us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 14:51:27.788865 (+ 950us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 14:51:27.788870 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 14:51:27.789551 (+ 681us) client_negotiation.cc:189] Negotiated authn=SASL
0620 14:51:27.789893 (+ 342us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:51:27.789910 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:51:27.792106 (+ 2196us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:51:27.792113 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:51:27.793470 (+ 1357us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:51:27.793487 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:51:27.794520 (+ 1033us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:51:27.794560 (+ 40us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:51:27.794771 (+ 211us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:51:27.795491 (+ 720us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 14:51:27.795514 (+ 23us) client_negotiation.cc:656] calling sasl_client_start()
0620 14:51:27.798241 (+ 2727us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 14:51:27.801639 (+ 3398us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 14:51:27.801657 (+ 18us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 14:51:27.801667 (+ 10us) client_negotiation.cc:760] calling sasl_client_step()
0620 14:51:27.802399 (+ 732us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 14:51:27.802811 (+ 412us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 14:51:27.802817 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 14:51:27.802820 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0620 14:51:27.802996 (+ 176us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 14:51:27.803679 (+ 683us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 14:51:27.803688 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 14:51:27.803908 (+ 220us) client_negotiation.cc:769] Sending connection context
0620 14:51:27.804124 (+ 216us) client_negotiation.cc:240] Negotiation successful
0620 14:51:27.804474 (+ 350us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1320,"thread_start_us":1238,"threads_started":1}
I20260620 14:51:27.806555 16586 tablet_service.cc:1514] Processing DeleteTablet for tablet f0b72f41b6e0405ba643578796791fa1 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 14:51:06 UTC) from {username='kudu', principal='kudu/127.0.97.62@KRBTEST.COM'} at 127.0.0.1:39910
I20260620 14:51:27.808830 16659 tablet_replica.cc:330] stopping tablet replica
I20260620 14:51:27.809253 16659 raft_consensus.cc:2229] T f0b72f41b6e0405ba643578796791fa1 P 531130dea5ed49f5926726011f1dc612 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 14:51:27.809662 16659 raft_consensus.cc:2258] T f0b72f41b6e0405ba643578796791fa1 P 531130dea5ed49f5926726011f1dc612 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 14:51:27.810989 16659 ts_tablet_manager.cc:1905] T f0b72f41b6e0405ba643578796791fa1 P 531130dea5ed49f5926726011f1dc612: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 14:51:27.818213 16659 ts_tablet_manager.cc:1918] T f0b72f41b6e0405ba643578796791fa1 P 531130dea5ed49f5926726011f1dc612: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260620 14:51:27.818432 16659 log.cc:1196] T f0b72f41b6e0405ba643578796791fa1 P 531130dea5ed49f5926726011f1dc612: Deleting WAL directory at /tmp/dist-test-taskUci3DU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781966829091300-388-0/minicluster-data/ts-0/wal/wals/f0b72f41b6e0405ba643578796791fa1
I20260620 14:51:27.818944 16659 ts_tablet_manager.cc:1939] T f0b72f41b6e0405ba643578796791fa1 P 531130dea5ed49f5926726011f1dc612: Deleting consensus metadata
I20260620 14:51:27.821478 16451 catalog_manager.cc:4841] TS 531130dea5ed49f5926726011f1dc612 (127.0.97.1:40009): tablet f0b72f41b6e0405ba643578796791fa1 (table test-table [id=3cff1ff4144a49c7adbe83aa6459f11e]) successfully deleted
W20260620 14:51:28.097652 16656 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 14:51:28.098012 16656 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 14:51:28.098179 16656 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 14:51:28.105835 16656 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 14:51:28.106011 16656 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 14:51:28.106197 16656 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.2
I20260620 14:51:28.116477 16656 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:44619
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskUci3DU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781966829091300-388-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskUci3DU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781966829091300-388-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskUci3DU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781966829091300-388-0/krb5kdc/kudu_127.0.97.2.keytab
--principal=kudu/127.0.97.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.2:32783
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskUci3DU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781966829091300-388-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.97.2
--webserver_port=40609
--tserver_master_addrs=127.0.97.62:45085
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.2
--log_dir=/tmp/dist-test-taskUci3DU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781966829091300-388-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 13:41:20 UTC on ccycloud.quasar-vdagoi.root.comops.site
build id 4457462
ASAN enabled
I20260620 14:51:28.117901 16656 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 14:51:28.119375 16656 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 14:51:28.133174 16664 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 14:51:28.133090 16665 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 14:51:28.133174 16667 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 14:51:28.134052 16656 server_base.cc:1034] running on GCE node
I20260620 14:51:28.134653 16656 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 14:51:28.135696 16656 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 14:51:28.136914 16656 hybrid_clock.cc:648] HybridClock initialized: now 1781967088136841 us; error 65 us; skew 500 ppm
I20260620 14:51:28.137266 16656 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 14:51:28 dist-test-slave-q9j6 krb5kdc[15879](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781967088, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 14:51:28.140563 16656 init.cc:377] Logged in from keytab as kudu/127.0.97.2@KRBTEST.COM (short username kudu)
I20260620 14:51:28.142249 16656 webserver.cc:460] Webserver started at http://127.0.97.2:40609/ using document root <none> and password file <none>
I20260620 14:51:28.143014 16656 fs_manager.cc:359] Metadata directory not provided
I20260620 14:51:28.143182 16656 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 14:51:28.149164 16656 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.001s sys 0.004s
I20260620 14:51:28.152294 16674 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 14:51:28.154066 16656 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260620 14:51:28.154269 16656 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskUci3DU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781966829091300-388-0/minicluster-data/ts-1/data,/tmp/dist-test-taskUci3DU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781966829091300-388-0/minicluster-data/ts-1/wal
uuid: "ff16403da3304c18aff565fade86842e"
format_stamp: "Formatted at 2026-06-20 14:51:05 on dist-test-slave-q9j6"
server_key: "028d91d47f18dd8924ae2d375a9553de"
server_key_iv: "f326979f277e705d2dc12dfd1218c78c"
server_key_version: "encryptionkey@0"
I20260620 14:51:28.155328 16656 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskUci3DU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781966829091300-388-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskUci3DU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781966829091300-388-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskUci3DU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781966829091300-388-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 14:51:28.181417 16656 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 14:51:28.185876 16656 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 14:51:28.186221 16656 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 14:51:28.187412 16656 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 14:51:28.188992 16656 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 14:51:28.189085 16656 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 14:51:28.189162 16656 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 14:51:28.189216 16656 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 14:51:28.227948 16656 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.2:32783
I20260620 14:51:28.228003 16786 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.2:32783 every 8 connection(s)
I20260620 14:51:28.229868 16656 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskUci3DU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781966829091300-388-0/minicluster-data/ts-1/data/info.pb
I20260620 14:51:28.232050 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskUci3DU/build/asan/bin/kudu as pid 16656
I20260620 14:51:28.232739 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskUci3DU/build/asan/bin/kudu
/tmp/dist-test-taskUci3DU/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskUci3DU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781966829091300-388-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskUci3DU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781966829091300-388-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskUci3DU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781966829091300-388-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskUci3DU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781966829091300-388-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.3:33429
--local_ip_for_outbound_sockets=127.0.97.3
--tserver_master_addrs=127.0.97.62:45085
--webserver_port=36959
--webserver_interface=127.0.97.3
--keytab_file=/tmp/dist-test-taskUci3DU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781966829091300-388-0/krb5kdc/kudu_127.0.97.3.keytab
--principal=kudu/127.0.97.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:44619
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskUci3DU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781966829091300-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskUci3DU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781966829091300-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskUci3DU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781966829091300-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Jun 20 14:51:28 dist-test-slave-q9j6 krb5kdc[15879](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781967088, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.2@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260620 14:51:28.252270 16653 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:51:28.232843 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.2:34017 (local address 127.0.97.62:45085)
0620 14:51:28.233054 (+ 211us) server_negotiation.cc:206] Beginning negotiation
0620 14:51:28.233062 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0620 14:51:28.234536 (+ 1474us) server_negotiation.cc:407] Connection header received
0620 14:51:28.236671 (+ 2135us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:51:28.236680 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:51:28.236802 (+ 122us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:51:28.236929 (+ 127us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:51:28.238614 (+ 1685us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:51:28.239684 (+ 1070us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:51:28.241410 (+ 1726us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:51:28.241853 (+ 443us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:51:28.242019 (+ 166us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:51:28.246192 (+ 4173us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 14:51:28.246197 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 14:51:28.246200 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 14:51:28.246237 (+ 37us) server_negotiation.cc:922] calling sasl_server_start()
0620 14:51:28.248988 (+ 2751us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:51:28.249826 (+ 838us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:51:28.249839 (+ 13us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:51:28.249843 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:51:28.249982 (+ 139us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:51:28.250440 (+ 458us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:51:28.250446 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:51:28.250451 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:51:28.250741 (+ 290us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 14:51:28.250903 (+ 162us) server_negotiation.cc:1035] Waiting for connection context
0620 14:51:28.251640 (+ 737us) server_negotiation.cc:299] Negotiation successful
0620 14:51:28.251908 (+ 268us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":85}
I20260620 14:51:28.253328 16789 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:51:28.233424 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:45085 (local address 127.0.97.2:34017)
0620 14:51:28.234309 (+ 885us) negotiation.cc:107] Waiting for socket to connect
0620 14:51:28.234368 (+ 59us) client_negotiation.cc:174] Beginning negotiation
0620 14:51:28.236325 (+ 1957us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 14:51:28.237166 (+ 841us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 14:51:28.237176 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 14:51:28.237771 (+ 595us) client_negotiation.cc:189] Negotiated authn=SASL
0620 14:51:28.238393 (+ 622us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:51:28.238412 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:51:28.239899 (+ 1487us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:51:28.239903 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:51:28.241221 (+ 1318us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:51:28.241238 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:51:28.242108 (+ 870us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:51:28.242118 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:51:28.242385 (+ 267us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:51:28.243191 (+ 806us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 14:51:28.243215 (+ 24us) client_negotiation.cc:656] calling sasl_client_start()
0620 14:51:28.245910 (+ 2695us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 14:51:28.249159 (+ 3249us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 14:51:28.249170 (+ 11us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 14:51:28.249191 (+ 21us) client_negotiation.cc:760] calling sasl_client_step()
0620 14:51:28.249665 (+ 474us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 14:51:28.250106 (+ 441us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 14:51:28.250111 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 14:51:28.250115 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0620 14:51:28.250321 (+ 206us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 14:51:28.250883 (+ 562us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 14:51:28.250891 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 14:51:28.251249 (+ 358us) client_negotiation.cc:769] Sending connection context
0620 14:51:28.251534 (+ 285us) client_negotiation.cc:240] Negotiation successful
0620 14:51:28.252160 (+ 626us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":573,"thread_start_us":377,"threads_started":1}
I20260620 14:51:28.254884 16787 heartbeater.cc:347] Connected to a master server at 127.0.97.62:45085
I20260620 14:51:28.255271 16787 heartbeater.cc:464] Registering TS with master...
I20260620 14:51:28.256304 16787 heartbeater.cc:511] Master 127.0.97.62:45085 requested a full tablet report, sending...
I20260620 14:51:28.258170 16464 ts_manager.cc:194] Registered new tserver with Master: ff16403da3304c18aff565fade86842e (127.0.97.2:32783)
I20260620 14:51:28.259346 16464 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.97.2@KRBTEST.COM'} at 127.0.97.2:34017
W20260620 14:51:28.576172 16790 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 14:51:28.576538 16790 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 14:51:28.576673 16790 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 14:51:28.584197 16790 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 14:51:28.584331 16790 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 14:51:28.584445 16790 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.3
I20260620 14:51:28.594561 16790 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:44619
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskUci3DU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781966829091300-388-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskUci3DU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781966829091300-388-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskUci3DU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781966829091300-388-0/krb5kdc/kudu_127.0.97.3.keytab
--principal=kudu/127.0.97.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.3:33429
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskUci3DU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781966829091300-388-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.97.3
--webserver_port=36959
--tserver_master_addrs=127.0.97.62:45085
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.3
--log_dir=/tmp/dist-test-taskUci3DU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781966829091300-388-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 13:41:20 UTC on ccycloud.quasar-vdagoi.root.comops.site
build id 4457462
ASAN enabled
I20260620 14:51:28.595928 16790 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 14:51:28.597334 16790 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 14:51:28.611617 16795 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 14:51:28.611553 16796 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 14:51:28.611464 16798 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 14:51:28.612344 16790 server_base.cc:1034] running on GCE node
I20260620 14:51:28.613147 16790 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 14:51:28.614558 16790 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 14:51:28.615816 16790 hybrid_clock.cc:648] HybridClock initialized: now 1781967088615746 us; error 52 us; skew 500 ppm
I20260620 14:51:28.616153 16790 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 14:51:28 dist-test-slave-q9j6 krb5kdc[15879](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781967088, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 14:51:28.619915 16790 init.cc:377] Logged in from keytab as kudu/127.0.97.3@KRBTEST.COM (short username kudu)
I20260620 14:51:28.622444 16790 webserver.cc:460] Webserver started at http://127.0.97.3:36959/ using document root <none> and password file <none>
I20260620 14:51:28.623340 16790 fs_manager.cc:359] Metadata directory not provided
I20260620 14:51:28.623505 16790 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 14:51:28.629145 16790 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.004s sys 0.000s
I20260620 14:51:28.631902 16805 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 14:51:28.633308 16790 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260620 14:51:28.633496 16790 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskUci3DU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781966829091300-388-0/minicluster-data/ts-2/data,/tmp/dist-test-taskUci3DU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781966829091300-388-0/minicluster-data/ts-2/wal
uuid: "6879ba8da72645c29edc506ff924ba31"
format_stamp: "Formatted at 2026-06-20 14:51:05 on dist-test-slave-q9j6"
server_key: "d36a4e4bd2f0abafa50e8a532e9e4c45"
server_key_iv: "bac9ca16940dc3b8aa0a4af457f0ce41"
server_key_version: "encryptionkey@0"
I20260620 14:51:28.634478 16790 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskUci3DU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781966829091300-388-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskUci3DU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781966829091300-388-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskUci3DU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781966829091300-388-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 14:51:28.649246 16790 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 14:51:28.654076 16790 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 14:51:28.654371 16790 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 14:51:28.655467 16790 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 14:51:28.656880 16790 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 14:51:28.656967 16790 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 14:51:28.657117 16790 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 14:51:28.657204 16790 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 14:51:28.692163 16790 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.3:33429
I20260620 14:51:28.692297 16917 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.3:33429 every 8 connection(s)
I20260620 14:51:28.693773 16790 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskUci3DU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781966829091300-388-0/minicluster-data/ts-2/data/info.pb
I20260620 14:51:28.699136 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskUci3DU/build/asan/bin/kudu as pid 16790
Jun 20 14:51:28 dist-test-slave-q9j6 krb5kdc[15879](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781967088, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.3@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260620 14:51:28.715708 16922 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:51:28.700192 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50146 (local address 127.0.97.62:45085)
0620 14:51:28.701496 (+ 1304us) server_negotiation.cc:206] Beginning negotiation
0620 14:51:28.701502 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 14:51:28.701523 (+ 21us) server_negotiation.cc:407] Connection header received
0620 14:51:28.701573 (+ 50us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:51:28.701578 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:51:28.701642 (+ 64us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:51:28.701767 (+ 125us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:51:28.703389 (+ 1622us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:51:28.704232 (+ 843us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:51:28.705624 (+ 1392us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:51:28.706180 (+ 556us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:51:28.706333 (+ 153us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:51:28.709268 (+ 2935us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 14:51:28.709275 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 14:51:28.709280 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 14:51:28.709330 (+ 50us) server_negotiation.cc:922] calling sasl_server_start()
0620 14:51:28.712728 (+ 3398us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:51:28.713669 (+ 941us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:51:28.713674 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:51:28.713677 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:51:28.713830 (+ 153us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:51:28.714231 (+ 401us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:51:28.714236 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:51:28.714239 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:51:28.714602 (+ 363us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 14:51:28.714882 (+ 280us) server_negotiation.cc:1035] Waiting for connection context
0620 14:51:28.715155 (+ 273us) server_negotiation.cc:299] Negotiation successful
0620 14:51:28.715342 (+ 187us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":46,"server-negotiator.queue_time_us":1149,"thread_start_us":963,"threads_started":1}
I20260620 14:51:28.717116 16653 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:51:28.696590 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.3:59051 (local address 127.0.97.62:45085)
0620 14:51:28.696785 (+ 195us) server_negotiation.cc:206] Beginning negotiation
0620 14:51:28.696790 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 14:51:28.698046 (+ 1256us) server_negotiation.cc:407] Connection header received
0620 14:51:28.700580 (+ 2534us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:51:28.700587 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:51:28.700681 (+ 94us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:51:28.700774 (+ 93us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:51:28.703123 (+ 2349us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:51:28.703935 (+ 812us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:51:28.705582 (+ 1647us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:51:28.706127 (+ 545us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:51:28.706265 (+ 138us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:51:28.710275 (+ 4010us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 14:51:28.710280 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 14:51:28.710283 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 14:51:28.710317 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0620 14:51:28.713416 (+ 3099us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:51:28.714405 (+ 989us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:51:28.714409 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:51:28.714411 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:51:28.714516 (+ 105us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:51:28.715093 (+ 577us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:51:28.715098 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:51:28.715101 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:51:28.715477 (+ 376us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 14:51:28.715662 (+ 185us) server_negotiation.cc:1035] Waiting for connection context
0620 14:51:28.716663 (+ 1001us) server_negotiation.cc:299] Negotiation successful
0620 14:51:28.716827 (+ 164us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":54,"server-negotiator.queue_time_us":65}
I20260620 14:51:28.718420 16920 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:51:28.697126 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:45085 (local address 127.0.97.3:59051)
0620 14:51:28.697867 (+ 741us) negotiation.cc:107] Waiting for socket to connect
0620 14:51:28.697906 (+ 39us) client_negotiation.cc:174] Beginning negotiation
0620 14:51:28.700198 (+ 2292us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 14:51:28.701522 (+ 1324us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 14:51:28.701538 (+ 16us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 14:51:28.702266 (+ 728us) client_negotiation.cc:189] Negotiated authn=SASL
0620 14:51:28.702912 (+ 646us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:51:28.702933 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:51:28.704119 (+ 1186us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:51:28.704123 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:51:28.705423 (+ 1300us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:51:28.705436 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:51:28.706373 (+ 937us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:51:28.706381 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:51:28.706655 (+ 274us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:51:28.707453 (+ 798us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 14:51:28.707478 (+ 25us) client_negotiation.cc:656] calling sasl_client_start()
0620 14:51:28.710041 (+ 2563us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 14:51:28.713591 (+ 3550us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 14:51:28.713627 (+ 36us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 14:51:28.713649 (+ 22us) client_negotiation.cc:760] calling sasl_client_step()
0620 14:51:28.714282 (+ 633us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 14:51:28.714651 (+ 369us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 14:51:28.714659 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 14:51:28.714664 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0620 14:51:28.714943 (+ 279us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 14:51:28.715658 (+ 715us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 14:51:28.715667 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 14:51:28.716173 (+ 506us) client_negotiation.cc:769] Sending connection context
0620 14:51:28.716563 (+ 390us) client_negotiation.cc:240] Negotiation successful
0620 14:51:28.717138 (+ 575us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":486,"thread_start_us":296,"threads_started":1}
I20260620 14:51:28.719964 16918 heartbeater.cc:347] Connected to a master server at 127.0.97.62:45085
I20260620 14:51:28.720273 16918 heartbeater.cc:464] Registering TS with master...
I20260620 14:51:28.721004 16918 heartbeater.cc:511] Master 127.0.97.62:45085 requested a full tablet report, sending...
I20260620 14:51:28.722433 16464 ts_manager.cc:194] Registered new tserver with Master: 6879ba8da72645c29edc506ff924ba31 (127.0.97.3:33429)
I20260620 14:51:28.723249 16464 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.97.3@KRBTEST.COM'} at 127.0.97.3:59051
I20260620 14:51:28.730916 388 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260620 14:51:28.737944 16653 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:51:28.731883 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50154 (local address 127.0.97.62:45085)
0620 14:51:28.732071 (+ 188us) server_negotiation.cc:206] Beginning negotiation
0620 14:51:28.732076 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 14:51:28.732463 (+ 387us) server_negotiation.cc:407] Connection header received
0620 14:51:28.732676 (+ 213us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:51:28.732679 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:51:28.732829 (+ 150us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:51:28.732966 (+ 137us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 14:51:28.733361 (+ 395us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:51:28.734111 (+ 750us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:51:28.735786 (+ 1675us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:51:28.736402 (+ 616us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:51:28.736526 (+ 124us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:51:28.737062 (+ 536us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 14:51:28.737305 (+ 243us) server_negotiation.cc:394] Sending RPC error: FATAL_INVALID_AUTHENTICATION_TOKEN: Not authorized: token expired
0620 14:51:28.737681 (+ 376us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:50154: token expired
Metrics: {"server-negotiator.queue_time_us":62}
W20260620 14:51:28.738101 16653 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:50154: token expired
W20260620 14:51:28.738582 16385 master_proxy_rpc.cc:203] Re-attempting CreateTable request to leader Master (127.0.97.62:45085)
I20260620 14:51:28.752800 16653 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:51:28.739154 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50168 (local address 127.0.97.62:45085)
0620 14:51:28.739320 (+ 166us) server_negotiation.cc:206] Beginning negotiation
0620 14:51:28.739326 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 14:51:28.739405 (+ 79us) server_negotiation.cc:407] Connection header received
0620 14:51:28.739566 (+ 161us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:51:28.739573 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:51:28.739696 (+ 123us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:51:28.739793 (+ 97us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:51:28.741220 (+ 1427us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:51:28.742284 (+ 1064us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:51:28.743800 (+ 1516us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:51:28.744441 (+ 641us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:51:28.744582 (+ 141us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:51:28.747363 (+ 2781us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 14:51:28.747370 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 14:51:28.747375 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 14:51:28.747425 (+ 50us) server_negotiation.cc:922] calling sasl_server_start()
0620 14:51:28.750144 (+ 2719us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:51:28.750931 (+ 787us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:51:28.750935 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:51:28.750937 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:51:28.751070 (+ 133us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:51:28.751417 (+ 347us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:51:28.751420 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:51:28.751422 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:51:28.751787 (+ 365us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 14:51:28.751992 (+ 205us) server_negotiation.cc:1035] Waiting for connection context
0620 14:51:28.752269 (+ 277us) server_negotiation.cc:299] Negotiation successful
0620 14:51:28.752518 (+ 249us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":59}
I20260620 14:51:28.756707 16464 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:50168:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260620 14:51:28.764736 16464 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260620 14:51:28.790838 16586 tablet_service.cc:1467] Processing CreateTablet for tablet 28b19f1b70ca4ec3951377191a874f51 (DEFAULT_TABLE table=test-table [id=1c2c084a4cfb4077aac1f9cc3032d813]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 14:51:28.791851 16586 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 28b19f1b70ca4ec3951377191a874f51. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 14:51:28.798794 16931 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:51:28.777130 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:32814 (local address 127.0.97.3:33429)
0620 14:51:28.788415 (+ 11285us) server_negotiation.cc:206] Beginning negotiation
0620 14:51:28.788431 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0620 14:51:28.789782 (+ 1351us) server_negotiation.cc:407] Connection header received
0620 14:51:28.790092 (+ 310us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:51:28.790101 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:51:28.790338 (+ 237us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:51:28.790503 (+ 165us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 14:51:28.791593 (+ 1090us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:51:28.793238 (+ 1645us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:51:28.796596 (+ 3358us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:51:28.797582 (+ 986us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:51:28.797767 (+ 185us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:51:28.798042 (+ 275us) server_negotiation.cc:1035] Waiting for connection context
0620 14:51:28.798292 (+ 250us) server_negotiation.cc:299] Negotiation successful
0620 14:51:28.798464 (+ 172us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":10981,"thread_start_us":11321,"threads_started":1}
I20260620 14:51:28.798832 16928 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:51:28.776721 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.3:33429 (local address 127.0.0.1:32814)
0620 14:51:28.789658 (+ 12937us) negotiation.cc:107] Waiting for socket to connect
0620 14:51:28.789677 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0620 14:51:28.789897 (+ 220us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 14:51:28.790859 (+ 962us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 14:51:28.790866 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 14:51:28.790900 (+ 34us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 14:51:28.791387 (+ 487us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:51:28.791403 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:51:28.793441 (+ 2038us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:51:28.793447 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:51:28.796417 (+ 2970us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:51:28.796434 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:51:28.797834 (+ 1400us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:51:28.797840 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:51:28.798019 (+ 179us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:51:28.798096 (+ 77us) client_negotiation.cc:769] Sending connection context
0620 14:51:28.798228 (+ 132us) client_negotiation.cc:240] Negotiation successful
0620 14:51:28.798390 (+ 162us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":12807,"mutex_wait_us":528,"thread_start_us":12237,"threads_started":1}
I20260620 14:51:28.800462 16930 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:51:28.776258 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.2:32783 (local address 127.0.0.1:42542)
0620 14:51:28.789361 (+ 13103us) negotiation.cc:107] Waiting for socket to connect
0620 14:51:28.789393 (+ 32us) client_negotiation.cc:174] Beginning negotiation
0620 14:51:28.789604 (+ 211us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 14:51:28.790677 (+ 1073us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 14:51:28.790685 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 14:51:28.790711 (+ 26us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 14:51:28.791208 (+ 497us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:51:28.791227 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:51:28.793441 (+ 2214us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:51:28.793446 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:51:28.796263 (+ 2817us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:51:28.796283 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:51:28.799586 (+ 3303us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:51:28.799620 (+ 34us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:51:28.799801 (+ 181us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:51:28.799835 (+ 34us) client_negotiation.cc:769] Sending connection context
0620 14:51:28.799927 (+ 92us) client_negotiation.cc:240] Negotiation successful
0620 14:51:28.800064 (+ 137us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":12913,"mutex_wait_us":107,"thread_start_us":13148,"threads_started":1}
I20260620 14:51:28.801170 16929 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:51:28.776525 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42542 (local address 127.0.97.2:32783)
0620 14:51:28.789341 (+ 12816us) server_negotiation.cc:206] Beginning negotiation
0620 14:51:28.789355 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0620 14:51:28.789615 (+ 260us) server_negotiation.cc:407] Connection header received
0620 14:51:28.789806 (+ 191us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:51:28.789817 (+ 11us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:51:28.790138 (+ 321us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:51:28.790362 (+ 224us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 14:51:28.791431 (+ 1069us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:51:28.793238 (+ 1807us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:51:28.796473 (+ 3235us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:51:28.798473 (+ 2000us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:51:28.798617 (+ 144us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:51:28.798834 (+ 217us) server_negotiation.cc:1035] Waiting for connection context
0620 14:51:28.800755 (+ 1921us) server_negotiation.cc:299] Negotiation successful
0620 14:51:28.800926 (+ 171us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":12591,"thread_start_us":12861,"threads_started":1}
I20260620 14:51:28.804149 16932 tablet_bootstrap.cc:492] T 28b19f1b70ca4ec3951377191a874f51 P 531130dea5ed49f5926726011f1dc612: Bootstrap starting.
I20260620 14:51:28.804643 16853 tablet_service.cc:1467] Processing CreateTablet for tablet 28b19f1b70ca4ec3951377191a874f51 (DEFAULT_TABLE table=test-table [id=1c2c084a4cfb4077aac1f9cc3032d813]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 14:51:28.805998 16853 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 28b19f1b70ca4ec3951377191a874f51. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 14:51:28.806035 16932 tablet_bootstrap.cc:654] T 28b19f1b70ca4ec3951377191a874f51 P 531130dea5ed49f5926726011f1dc612: Neither blocks nor log segments found. Creating new log.
I20260620 14:51:28.808212 16932 tablet_bootstrap.cc:492] T 28b19f1b70ca4ec3951377191a874f51 P 531130dea5ed49f5926726011f1dc612: No bootstrap required, opened a new log
I20260620 14:51:28.808378 16932 ts_tablet_manager.cc:1397] T 28b19f1b70ca4ec3951377191a874f51 P 531130dea5ed49f5926726011f1dc612: Time spent bootstrapping tablet: real 0.004s user 0.003s sys 0.000s
I20260620 14:51:28.809085 16932 raft_consensus.cc:348] T 28b19f1b70ca4ec3951377191a874f51 P 531130dea5ed49f5926726011f1dc612 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ff16403da3304c18aff565fade86842e" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 32783 } } peers { permanent_uuid: "6879ba8da72645c29edc506ff924ba31" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 33429 } } peers { permanent_uuid: "531130dea5ed49f5926726011f1dc612" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 40009 } }
I20260620 14:51:28.809283 16932 raft_consensus.cc:374] T 28b19f1b70ca4ec3951377191a874f51 P 531130dea5ed49f5926726011f1dc612 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 14:51:28.809404 16932 raft_consensus.cc:729] T 28b19f1b70ca4ec3951377191a874f51 P 531130dea5ed49f5926726011f1dc612 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 531130dea5ed49f5926726011f1dc612, State: Initialized, Role: FOLLOWER
I20260620 14:51:28.809688 16932 consensus_queue.cc:260] T 28b19f1b70ca4ec3951377191a874f51 P 531130dea5ed49f5926726011f1dc612 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ff16403da3304c18aff565fade86842e" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 32783 } } peers { permanent_uuid: "6879ba8da72645c29edc506ff924ba31" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 33429 } } peers { permanent_uuid: "531130dea5ed49f5926726011f1dc612" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 40009 } }
I20260620 14:51:28.810488 16932 ts_tablet_manager.cc:1428] T 28b19f1b70ca4ec3951377191a874f51 P 531130dea5ed49f5926726011f1dc612: Time spent starting tablet: real 0.002s user 0.003s sys 0.000s
I20260620 14:51:28.816779 16722 tablet_service.cc:1467] Processing CreateTablet for tablet 28b19f1b70ca4ec3951377191a874f51 (DEFAULT_TABLE table=test-table [id=1c2c084a4cfb4077aac1f9cc3032d813]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 14:51:28.818449 16722 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 28b19f1b70ca4ec3951377191a874f51. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 14:51:28.822701 16935 tablet_bootstrap.cc:492] T 28b19f1b70ca4ec3951377191a874f51 P 6879ba8da72645c29edc506ff924ba31: Bootstrap starting.
I20260620 14:51:28.826499 16935 tablet_bootstrap.cc:654] T 28b19f1b70ca4ec3951377191a874f51 P 6879ba8da72645c29edc506ff924ba31: Neither blocks nor log segments found. Creating new log.
I20260620 14:51:28.827662 16935 log.cc:824] T 28b19f1b70ca4ec3951377191a874f51 P 6879ba8da72645c29edc506ff924ba31: Log is configured to *not* fsync() on all Append() calls
I20260620 14:51:28.830652 16935 tablet_bootstrap.cc:492] T 28b19f1b70ca4ec3951377191a874f51 P 6879ba8da72645c29edc506ff924ba31: No bootstrap required, opened a new log
I20260620 14:51:28.831022 16935 ts_tablet_manager.cc:1397] T 28b19f1b70ca4ec3951377191a874f51 P 6879ba8da72645c29edc506ff924ba31: Time spent bootstrapping tablet: real 0.009s user 0.003s sys 0.003s
I20260620 14:51:28.833801 16937 tablet_bootstrap.cc:492] T 28b19f1b70ca4ec3951377191a874f51 P ff16403da3304c18aff565fade86842e: Bootstrap starting.
I20260620 14:51:28.837217 16937 tablet_bootstrap.cc:654] T 28b19f1b70ca4ec3951377191a874f51 P ff16403da3304c18aff565fade86842e: Neither blocks nor log segments found. Creating new log.
I20260620 14:51:28.838599 16937 log.cc:824] T 28b19f1b70ca4ec3951377191a874f51 P ff16403da3304c18aff565fade86842e: Log is configured to *not* fsync() on all Append() calls
I20260620 14:51:28.838975 16935 raft_consensus.cc:348] T 28b19f1b70ca4ec3951377191a874f51 P 6879ba8da72645c29edc506ff924ba31 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ff16403da3304c18aff565fade86842e" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 32783 } } peers { permanent_uuid: "6879ba8da72645c29edc506ff924ba31" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 33429 } } peers { permanent_uuid: "531130dea5ed49f5926726011f1dc612" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 40009 } }
I20260620 14:51:28.839439 16935 raft_consensus.cc:374] T 28b19f1b70ca4ec3951377191a874f51 P 6879ba8da72645c29edc506ff924ba31 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 14:51:28.839550 16935 raft_consensus.cc:729] T 28b19f1b70ca4ec3951377191a874f51 P 6879ba8da72645c29edc506ff924ba31 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6879ba8da72645c29edc506ff924ba31, State: Initialized, Role: FOLLOWER
I20260620 14:51:28.840248 16935 consensus_queue.cc:260] T 28b19f1b70ca4ec3951377191a874f51 P 6879ba8da72645c29edc506ff924ba31 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ff16403da3304c18aff565fade86842e" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 32783 } } peers { permanent_uuid: "6879ba8da72645c29edc506ff924ba31" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 33429 } } peers { permanent_uuid: "531130dea5ed49f5926726011f1dc612" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 40009 } }
I20260620 14:51:28.841982 16918 heartbeater.cc:503] Master 127.0.97.62:45085 was elected leader, sending a full tablet report...
I20260620 14:51:28.842298 16937 tablet_bootstrap.cc:492] T 28b19f1b70ca4ec3951377191a874f51 P ff16403da3304c18aff565fade86842e: No bootstrap required, opened a new log
I20260620 14:51:28.842609 16937 ts_tablet_manager.cc:1397] T 28b19f1b70ca4ec3951377191a874f51 P ff16403da3304c18aff565fade86842e: Time spent bootstrapping tablet: real 0.009s user 0.006s sys 0.000s
I20260620 14:51:28.842630 16935 ts_tablet_manager.cc:1428] T 28b19f1b70ca4ec3951377191a874f51 P 6879ba8da72645c29edc506ff924ba31: Time spent starting tablet: real 0.011s user 0.008s sys 0.004s
I20260620 14:51:28.851784 16937 raft_consensus.cc:348] T 28b19f1b70ca4ec3951377191a874f51 P ff16403da3304c18aff565fade86842e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ff16403da3304c18aff565fade86842e" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 32783 } } peers { permanent_uuid: "6879ba8da72645c29edc506ff924ba31" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 33429 } } peers { permanent_uuid: "531130dea5ed49f5926726011f1dc612" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 40009 } }
I20260620 14:51:28.852394 16937 raft_consensus.cc:374] T 28b19f1b70ca4ec3951377191a874f51 P ff16403da3304c18aff565fade86842e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 14:51:28.852547 16937 raft_consensus.cc:729] T 28b19f1b70ca4ec3951377191a874f51 P ff16403da3304c18aff565fade86842e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ff16403da3304c18aff565fade86842e, State: Initialized, Role: FOLLOWER
I20260620 14:51:28.853224 16937 consensus_queue.cc:260] T 28b19f1b70ca4ec3951377191a874f51 P ff16403da3304c18aff565fade86842e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ff16403da3304c18aff565fade86842e" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 32783 } } peers { permanent_uuid: "6879ba8da72645c29edc506ff924ba31" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 33429 } } peers { permanent_uuid: "531130dea5ed49f5926726011f1dc612" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 40009 } }
I20260620 14:51:28.854274 16787 heartbeater.cc:503] Master 127.0.97.62:45085 was elected leader, sending a full tablet report...
I20260620 14:51:28.854946 16937 ts_tablet_manager.cc:1428] T 28b19f1b70ca4ec3951377191a874f51 P ff16403da3304c18aff565fade86842e: Time spent starting tablet: real 0.012s user 0.009s sys 0.005s
W20260620 14:51:28.945560 16919 tablet.cc:2369] T 28b19f1b70ca4ec3951377191a874f51 P 6879ba8da72645c29edc506ff924ba31: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260620 14:51:28.982172 16788 tablet.cc:2369] T 28b19f1b70ca4ec3951377191a874f51 P ff16403da3304c18aff565fade86842e: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260620 14:51:28.990128 16652 tablet.cc:2369] T 28b19f1b70ca4ec3951377191a874f51 P 531130dea5ed49f5926726011f1dc612: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260620 14:51:29.120400 16940 raft_consensus.cc:482] T 28b19f1b70ca4ec3951377191a874f51 P ff16403da3304c18aff565fade86842e [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260620 14:51:29.120744 16940 raft_consensus.cc:504] T 28b19f1b70ca4ec3951377191a874f51 P ff16403da3304c18aff565fade86842e [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ff16403da3304c18aff565fade86842e" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 32783 } } peers { permanent_uuid: "6879ba8da72645c29edc506ff924ba31" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 33429 } } peers { permanent_uuid: "531130dea5ed49f5926726011f1dc612" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 40009 } }
I20260620 14:51:29.131193 16940 leader_election.cc:288] T 28b19f1b70ca4ec3951377191a874f51 P ff16403da3304c18aff565fade86842e [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 6879ba8da72645c29edc506ff924ba31 (127.0.97.3:33429), 531130dea5ed49f5926726011f1dc612 (127.0.97.1:40009)
I20260620 14:51:29.137992 16931 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:51:29.127885 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.2:54025 (local address 127.0.97.3:33429)
0620 14:51:29.128615 (+ 730us) server_negotiation.cc:206] Beginning negotiation
0620 14:51:29.128622 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0620 14:51:29.128682 (+ 60us) server_negotiation.cc:407] Connection header received
0620 14:51:29.128763 (+ 81us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:51:29.128772 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:51:29.128860 (+ 88us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:51:29.128961 (+ 101us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 14:51:29.129668 (+ 707us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:51:29.130817 (+ 1149us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:51:29.135688 (+ 4871us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:51:29.136818 (+ 1130us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:51:29.136968 (+ 150us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:51:29.137112 (+ 144us) server_negotiation.cc:1035] Waiting for connection context
0620 14:51:29.137579 (+ 467us) server_negotiation.cc:299] Negotiation successful
0620 14:51:29.137729 (+ 150us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":539}
I20260620 14:51:29.138161 16941 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:51:29.122879 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.3:33429 (local address 127.0.97.2:54025)
0620 14:51:29.128408 (+ 5529us) negotiation.cc:107] Waiting for socket to connect
0620 14:51:29.128430 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0620 14:51:29.128585 (+ 155us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 14:51:29.128994 (+ 409us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 14:51:29.129000 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 14:51:29.129029 (+ 29us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 14:51:29.129504 (+ 475us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:51:29.129520 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:51:29.130980 (+ 1460us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:51:29.130991 (+ 11us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:51:29.135456 (+ 4465us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:51:29.135473 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:51:29.137107 (+ 1634us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:51:29.137118 (+ 11us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:51:29.137357 (+ 239us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:51:29.137408 (+ 51us) client_negotiation.cc:769] Sending connection context
0620 14:51:29.137520 (+ 112us) client_negotiation.cc:240] Negotiation successful
0620 14:51:29.137710 (+ 190us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":5368,"thread_start_us":5157,"threads_started":1}
I20260620 14:51:29.139459 16873 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "28b19f1b70ca4ec3951377191a874f51" candidate_uuid: "ff16403da3304c18aff565fade86842e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6879ba8da72645c29edc506ff924ba31" is_pre_election: true
I20260620 14:51:29.140110 16873 raft_consensus.cc:2454] T 28b19f1b70ca4ec3951377191a874f51 P 6879ba8da72645c29edc506ff924ba31 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ff16403da3304c18aff565fade86842e in term 0.
I20260620 14:51:29.140965 16676 leader_election.cc:302] T 28b19f1b70ca4ec3951377191a874f51 P ff16403da3304c18aff565fade86842e [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6879ba8da72645c29edc506ff924ba31, ff16403da3304c18aff565fade86842e; no voters:
I20260620 14:51:29.141376 16943 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:51:29.129631 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.2:43757 (local address 127.0.97.1:40009)
0620 14:51:29.130252 (+ 621us) server_negotiation.cc:206] Beginning negotiation
0620 14:51:29.130261 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0620 14:51:29.134203 (+ 3942us) server_negotiation.cc:407] Connection header received
0620 14:51:29.134287 (+ 84us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:51:29.134293 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:51:29.134385 (+ 92us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:51:29.134506 (+ 121us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 14:51:29.135349 (+ 843us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:51:29.136667 (+ 1318us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:51:29.139183 (+ 2516us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:51:29.140247 (+ 1064us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:51:29.140409 (+ 162us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:51:29.140670 (+ 261us) server_negotiation.cc:1035] Waiting for connection context
0620 14:51:29.140900 (+ 230us) server_negotiation.cc:299] Negotiation successful
0620 14:51:29.141040 (+ 140us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":442,"thread_start_us":310,"threads_started":1}
I20260620 14:51:29.141445 16940 raft_consensus.cc:2781] T 28b19f1b70ca4ec3951377191a874f51 P ff16403da3304c18aff565fade86842e [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260620 14:51:29.141592 16940 raft_consensus.cc:482] T 28b19f1b70ca4ec3951377191a874f51 P ff16403da3304c18aff565fade86842e [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260620 14:51:29.141745 16940 raft_consensus.cc:3037] T 28b19f1b70ca4ec3951377191a874f51 P ff16403da3304c18aff565fade86842e [term 0 FOLLOWER]: Advancing to term 1
I20260620 14:51:29.143844 16942 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:51:29.128890 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:40009 (local address 127.0.97.2:43757)
0620 14:51:29.133904 (+ 5014us) negotiation.cc:107] Waiting for socket to connect
0620 14:51:29.133927 (+ 23us) client_negotiation.cc:174] Beginning negotiation
0620 14:51:29.134073 (+ 146us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 14:51:29.134566 (+ 493us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 14:51:29.134571 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 14:51:29.134610 (+ 39us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 14:51:29.135113 (+ 503us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:51:29.135129 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:51:29.136836 (+ 1707us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:51:29.136841 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:51:29.139009 (+ 2168us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:51:29.139023 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:51:29.140514 (+ 1491us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:51:29.140520 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:51:29.140724 (+ 204us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:51:29.140760 (+ 36us) client_negotiation.cc:769] Sending connection context
0620 14:51:29.140870 (+ 110us) client_negotiation.cc:240] Negotiation successful
0620 14:51:29.141018 (+ 148us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":4881,"thread_start_us":2141,"threads_started":1}
I20260620 14:51:29.144204 16940 raft_consensus.cc:504] T 28b19f1b70ca4ec3951377191a874f51 P ff16403da3304c18aff565fade86842e [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ff16403da3304c18aff565fade86842e" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 32783 } } peers { permanent_uuid: "6879ba8da72645c29edc506ff924ba31" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 33429 } } peers { permanent_uuid: "531130dea5ed49f5926726011f1dc612" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 40009 } }
I20260620 14:51:29.145020 16606 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "28b19f1b70ca4ec3951377191a874f51" candidate_uuid: "ff16403da3304c18aff565fade86842e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "531130dea5ed49f5926726011f1dc612" is_pre_election: true
I20260620 14:51:29.145321 16940 leader_election.cc:288] T 28b19f1b70ca4ec3951377191a874f51 P ff16403da3304c18aff565fade86842e [CANDIDATE]: Term 1 election: Requested vote from peers 6879ba8da72645c29edc506ff924ba31 (127.0.97.3:33429), 531130dea5ed49f5926726011f1dc612 (127.0.97.1:40009)
I20260620 14:51:29.145437 16873 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "28b19f1b70ca4ec3951377191a874f51" candidate_uuid: "ff16403da3304c18aff565fade86842e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6879ba8da72645c29edc506ff924ba31"
I20260620 14:51:29.145470 16605 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "28b19f1b70ca4ec3951377191a874f51" candidate_uuid: "ff16403da3304c18aff565fade86842e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "531130dea5ed49f5926726011f1dc612"
I20260620 14:51:29.145624 16873 raft_consensus.cc:3037] T 28b19f1b70ca4ec3951377191a874f51 P 6879ba8da72645c29edc506ff924ba31 [term 0 FOLLOWER]: Advancing to term 1
I20260620 14:51:29.145677 16606 raft_consensus.cc:2454] T 28b19f1b70ca4ec3951377191a874f51 P 531130dea5ed49f5926726011f1dc612 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ff16403da3304c18aff565fade86842e in term 0.
I20260620 14:51:29.148304 16873 raft_consensus.cc:2454] T 28b19f1b70ca4ec3951377191a874f51 P 6879ba8da72645c29edc506ff924ba31 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ff16403da3304c18aff565fade86842e in term 1.
I20260620 14:51:29.148761 16676 leader_election.cc:302] T 28b19f1b70ca4ec3951377191a874f51 P ff16403da3304c18aff565fade86842e [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 6879ba8da72645c29edc506ff924ba31, ff16403da3304c18aff565fade86842e; no voters: 531130dea5ed49f5926726011f1dc612
I20260620 14:51:29.149111 16940 raft_consensus.cc:2781] T 28b19f1b70ca4ec3951377191a874f51 P ff16403da3304c18aff565fade86842e [term 1 FOLLOWER]: Leader election won for term 1
I20260620 14:51:29.149632 16940 raft_consensus.cc:686] T 28b19f1b70ca4ec3951377191a874f51 P ff16403da3304c18aff565fade86842e [term 1 LEADER]: Becoming Leader. State: Replica: ff16403da3304c18aff565fade86842e, State: Running, Role: LEADER
I20260620 14:51:29.150209 16940 consensus_queue.cc:237] T 28b19f1b70ca4ec3951377191a874f51 P ff16403da3304c18aff565fade86842e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ff16403da3304c18aff565fade86842e" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 32783 } } peers { permanent_uuid: "6879ba8da72645c29edc506ff924ba31" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 33429 } } peers { permanent_uuid: "531130dea5ed49f5926726011f1dc612" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 40009 } }
I20260620 14:51:29.155952 16464 catalog_manager.cc:5495] T 28b19f1b70ca4ec3951377191a874f51 P ff16403da3304c18aff565fade86842e reported cstate change: term changed from 0 to 1, leader changed from <none> to ff16403da3304c18aff565fade86842e (127.0.97.2). New cstate: current_term: 1 leader_uuid: "ff16403da3304c18aff565fade86842e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ff16403da3304c18aff565fade86842e" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 32783 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "6879ba8da72645c29edc506ff924ba31" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 33429 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "531130dea5ed49f5926726011f1dc612" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 40009 } health_report { overall_health: UNKNOWN } } }
I20260620 14:51:29.181279 16929 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:51:29.175463 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42544 (local address 127.0.97.2:32783)
0620 14:51:29.175724 (+ 261us) server_negotiation.cc:206] Beginning negotiation
0620 14:51:29.175731 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0620 14:51:29.175805 (+ 74us) server_negotiation.cc:407] Connection header received
0620 14:51:29.176063 (+ 258us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:51:29.176067 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:51:29.176158 (+ 91us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:51:29.176327 (+ 169us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 14:51:29.176701 (+ 374us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:51:29.177422 (+ 721us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:51:29.178847 (+ 1425us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:51:29.179281 (+ 434us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:51:29.179423 (+ 142us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:51:29.180076 (+ 653us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 14:51:29.180643 (+ 567us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0620 14:51:29.180822 (+ 179us) server_negotiation.cc:1035] Waiting for connection context
0620 14:51:29.180889 (+ 67us) server_negotiation.cc:299] Negotiation successful
0620 14:51:29.180980 (+ 91us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":82}
I20260620 14:51:29.190157 16605 raft_consensus.cc:3037] T 28b19f1b70ca4ec3951377191a874f51 P 531130dea5ed49f5926726011f1dc612 [term 0 FOLLOWER]: Advancing to term 1
I20260620 14:51:29.190482 16873 raft_consensus.cc:1261] T 28b19f1b70ca4ec3951377191a874f51 P 6879ba8da72645c29edc506ff924ba31 [term 1 FOLLOWER]: Refusing update from remote peer ff16403da3304c18aff565fade86842e: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 14:51:29.191421 16944 consensus_queue.cc:1035] T 28b19f1b70ca4ec3951377191a874f51 P ff16403da3304c18aff565fade86842e [LEADER]: Connected to new peer: Peer: permanent_uuid: "6879ba8da72645c29edc506ff924ba31" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 33429 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 14:51:29.193481 16605 raft_consensus.cc:1261] T 28b19f1b70ca4ec3951377191a874f51 P 531130dea5ed49f5926726011f1dc612 [term 1 FOLLOWER]: Refusing update from remote peer ff16403da3304c18aff565fade86842e: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 14:51:29.194298 16944 consensus_queue.cc:1035] T 28b19f1b70ca4ec3951377191a874f51 P ff16403da3304c18aff565fade86842e [LEADER]: Connected to new peer: Peer: permanent_uuid: "531130dea5ed49f5926726011f1dc612" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 40009 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 14:51:29.203984 16947 mvcc.cc:205] Tried to move back new op lower bound from 7298937197313110016 to 7298937197164437504. Current Snapshot: MvccSnapshot[applied={T|T < 7298937197313110016}]
I20260620 14:51:29.237507 16464 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:50168:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 14:51:29.237795 16464 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:50168:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 14:51:29.242169 16464 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 0069e3cceda94cd68d41c5833a5174c9: Sending DeleteTablet for 3 replicas of tablet 28b19f1b70ca4ec3951377191a874f51
I20260620 14:51:29.243311 16853 tablet_service.cc:1514] Processing DeleteTablet for tablet 28b19f1b70ca4ec3951377191a874f51 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 14:51:29 UTC) from {username='kudu', principal='kudu/127.0.97.62@KRBTEST.COM'} at 127.0.0.1:32814
I20260620 14:51:29.243327 16722 tablet_service.cc:1514] Processing DeleteTablet for tablet 28b19f1b70ca4ec3951377191a874f51 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 14:51:29 UTC) from {username='kudu', principal='kudu/127.0.97.62@KRBTEST.COM'} at 127.0.0.1:42542
I20260620 14:51:29.244458 16958 tablet_replica.cc:330] T 28b19f1b70ca4ec3951377191a874f51 P ff16403da3304c18aff565fade86842e: stopping tablet replica
I20260620 14:51:29.245038 16958 raft_consensus.cc:2229] T 28b19f1b70ca4ec3951377191a874f51 P ff16403da3304c18aff565fade86842e [term 1 LEADER]: Raft consensus shutting down.
I20260620 14:51:29.245059 16586 tablet_service.cc:1514] Processing DeleteTablet for tablet 28b19f1b70ca4ec3951377191a874f51 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 14:51:29 UTC) from {username='kudu', principal='kudu/127.0.97.62@KRBTEST.COM'} at 127.0.0.1:39910
I20260620 14:51:29.245937 16958 raft_consensus.cc:2258] T 28b19f1b70ca4ec3951377191a874f51 P ff16403da3304c18aff565fade86842e [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 14:51:29.246820 16957 tablet_replica.cc:330] T 28b19f1b70ca4ec3951377191a874f51 P 6879ba8da72645c29edc506ff924ba31: stopping tablet replica
I20260620 14:51:29.247367 16957 raft_consensus.cc:2229] T 28b19f1b70ca4ec3951377191a874f51 P 6879ba8da72645c29edc506ff924ba31 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 14:51:29.247993 16957 raft_consensus.cc:2258] T 28b19f1b70ca4ec3951377191a874f51 P 6879ba8da72645c29edc506ff924ba31 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 14:51:29.248103 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskUci3DU/build/asan/bin/kudu pid 16501
I20260620 14:51:29.250504 16957 ts_tablet_manager.cc:1905] T 28b19f1b70ca4ec3951377191a874f51 P 6879ba8da72645c29edc506ff924ba31: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 14:51:29.253717 16959 tablet_replica.cc:330] T 28b19f1b70ca4ec3951377191a874f51 P 531130dea5ed49f5926726011f1dc612: stopping tablet replica
I20260620 14:51:29.253952 16959 raft_consensus.cc:2229] T 28b19f1b70ca4ec3951377191a874f51 P 531130dea5ed49f5926726011f1dc612 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 14:51:29.254333 16959 raft_consensus.cc:2258] T 28b19f1b70ca4ec3951377191a874f51 P 531130dea5ed49f5926726011f1dc612 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 14:51:29.256348 16959 ts_tablet_manager.cc:1905] T 28b19f1b70ca4ec3951377191a874f51 P 531130dea5ed49f5926726011f1dc612: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 14:51:29.257480 16958 ts_tablet_manager.cc:1905] T 28b19f1b70ca4ec3951377191a874f51 P ff16403da3304c18aff565fade86842e: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 14:51:29.259472 16957 ts_tablet_manager.cc:1918] T 28b19f1b70ca4ec3951377191a874f51 P 6879ba8da72645c29edc506ff924ba31: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260620 14:51:29.259800 16957 log.cc:1196] T 28b19f1b70ca4ec3951377191a874f51 P 6879ba8da72645c29edc506ff924ba31: Deleting WAL directory at /tmp/dist-test-taskUci3DU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781966829091300-388-0/minicluster-data/ts-2/wal/wals/28b19f1b70ca4ec3951377191a874f51
I20260620 14:51:29.260512 16957 ts_tablet_manager.cc:1939] T 28b19f1b70ca4ec3951377191a874f51 P 6879ba8da72645c29edc506ff924ba31: Deleting consensus metadata
I20260620 14:51:29.262594 16449 catalog_manager.cc:4841] TS 6879ba8da72645c29edc506ff924ba31 (127.0.97.3:33429): tablet 28b19f1b70ca4ec3951377191a874f51 (table test-table [id=1c2c084a4cfb4077aac1f9cc3032d813]) successfully deleted
I20260620 14:51:29.267714 16958 ts_tablet_manager.cc:1918] T 28b19f1b70ca4ec3951377191a874f51 P ff16403da3304c18aff565fade86842e: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260620 14:51:29.267926 16958 log.cc:1196] T 28b19f1b70ca4ec3951377191a874f51 P ff16403da3304c18aff565fade86842e: Deleting WAL directory at /tmp/dist-test-taskUci3DU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781966829091300-388-0/minicluster-data/ts-1/wal/wals/28b19f1b70ca4ec3951377191a874f51
I20260620 14:51:29.268534 16958 ts_tablet_manager.cc:1939] T 28b19f1b70ca4ec3951377191a874f51 P ff16403da3304c18aff565fade86842e: Deleting consensus metadata
I20260620 14:51:29.269740 16959 ts_tablet_manager.cc:1918] T 28b19f1b70ca4ec3951377191a874f51 P 531130dea5ed49f5926726011f1dc612: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260620 14:51:29.269898 16959 log.cc:1196] T 28b19f1b70ca4ec3951377191a874f51 P 531130dea5ed49f5926726011f1dc612: Deleting WAL directory at /tmp/dist-test-taskUci3DU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781966829091300-388-0/minicluster-data/ts-0/wal/wals/28b19f1b70ca4ec3951377191a874f51
I20260620 14:51:29.270350 16959 ts_tablet_manager.cc:1939] T 28b19f1b70ca4ec3951377191a874f51 P 531130dea5ed49f5926726011f1dc612: Deleting consensus metadata
I20260620 14:51:29.270653 16451 catalog_manager.cc:4841] TS ff16403da3304c18aff565fade86842e (127.0.97.2:32783): tablet 28b19f1b70ca4ec3951377191a874f51 (table test-table [id=1c2c084a4cfb4077aac1f9cc3032d813]) successfully deleted
I20260620 14:51:29.271482 16451 catalog_manager.cc:4841] TS 531130dea5ed49f5926726011f1dc612 (127.0.97.1:40009): tablet 28b19f1b70ca4ec3951377191a874f51 (table test-table [id=1c2c084a4cfb4077aac1f9cc3032d813]) successfully deleted
I20260620 14:51:29.282100 16943 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:51:29.249248 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39922 (local address 127.0.97.1:40009)
0620 14:51:29.249462 (+ 214us) server_negotiation.cc:206] Beginning negotiation
0620 14:51:29.249468 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 14:51:29.255406 (+ 5938us) server_negotiation.cc:407] Connection header received
0620 14:51:29.255492 (+ 86us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:51:29.255500 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:51:29.255631 (+ 131us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:51:29.255751 (+ 120us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:51:29.264910 (+ 9159us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:51:29.265959 (+ 1049us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:51:29.272784 (+ 6825us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:51:29.273288 (+ 504us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:51:29.273447 (+ 159us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:51:29.276325 (+ 2878us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 14:51:29.276330 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 14:51:29.276334 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 14:51:29.276374 (+ 40us) server_negotiation.cc:922] calling sasl_server_start()
0620 14:51:29.279038 (+ 2664us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:51:29.279942 (+ 904us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:51:29.279950 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:51:29.279955 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:51:29.280166 (+ 211us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:51:29.280577 (+ 411us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:51:29.280582 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:51:29.280585 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:51:29.280925 (+ 340us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 14:51:29.281209 (+ 284us) server_negotiation.cc:1035] Waiting for connection context
0620 14:51:29.281457 (+ 248us) server_negotiation.cc:299] Negotiation successful
0620 14:51:29.281713 (+ 256us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":69}
I20260620 14:51:29.282784 16646 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 14:51:29.379101 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskUci3DU/build/asan/bin/kudu with pid 16501
I20260620 14:51:29.401845 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskUci3DU/build/asan/bin/kudu pid 16656
I20260620 14:51:29.417411 16929 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:51:29.402738 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42558 (local address 127.0.97.2:32783)
0620 14:51:29.402924 (+ 186us) server_negotiation.cc:206] Beginning negotiation
0620 14:51:29.402932 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0620 14:51:29.402999 (+ 67us) server_negotiation.cc:407] Connection header received
0620 14:51:29.403216 (+ 217us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:51:29.403222 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:51:29.403310 (+ 88us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:51:29.403434 (+ 124us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:51:29.404957 (+ 1523us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:51:29.405786 (+ 829us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:51:29.407412 (+ 1626us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:51:29.407948 (+ 536us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:51:29.408101 (+ 153us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:51:29.410949 (+ 2848us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 14:51:29.410966 (+ 17us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 14:51:29.410984 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 14:51:29.411028 (+ 44us) server_negotiation.cc:922] calling sasl_server_start()
0620 14:51:29.414238 (+ 3210us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:51:29.415098 (+ 860us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:51:29.415106 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:51:29.415109 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:51:29.415269 (+ 160us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:51:29.415750 (+ 481us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:51:29.415756 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:51:29.415759 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:51:29.416138 (+ 379us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 14:51:29.416518 (+ 380us) server_negotiation.cc:1035] Waiting for connection context
0620 14:51:29.416774 (+ 256us) server_negotiation.cc:299] Negotiation successful
0620 14:51:29.417008 (+ 234us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":75}
I20260620 14:51:29.418191 16782 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 14:51:29.534339 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskUci3DU/build/asan/bin/kudu with pid 16656
I20260620 14:51:29.553130 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskUci3DU/build/asan/bin/kudu pid 16790
I20260620 14:51:29.570430 16931 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:51:29.554017 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60024 (local address 127.0.97.3:33429)
0620 14:51:29.554230 (+ 213us) server_negotiation.cc:206] Beginning negotiation
0620 14:51:29.554235 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 14:51:29.554254 (+ 19us) server_negotiation.cc:407] Connection header received
0620 14:51:29.554414 (+ 160us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:51:29.554418 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:51:29.554494 (+ 76us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:51:29.554582 (+ 88us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:51:29.555941 (+ 1359us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:51:29.556658 (+ 717us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:51:29.558035 (+ 1377us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:51:29.558410 (+ 375us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:51:29.558504 (+ 94us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:51:29.564799 (+ 6295us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 14:51:29.564816 (+ 17us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 14:51:29.564833 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 14:51:29.564872 (+ 39us) server_negotiation.cc:922] calling sasl_server_start()
0620 14:51:29.567871 (+ 2999us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:51:29.568543 (+ 672us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:51:29.568550 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:51:29.568552 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:51:29.568701 (+ 149us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:51:29.569023 (+ 322us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:51:29.569026 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:51:29.569028 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:51:29.569382 (+ 354us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 14:51:29.569604 (+ 222us) server_negotiation.cc:1035] Waiting for connection context
0620 14:51:29.569865 (+ 261us) server_negotiation.cc:299] Negotiation successful
0620 14:51:29.570034 (+ 169us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":76}
I20260620 14:51:29.571085 16913 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 14:51:29.662425 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskUci3DU/build/asan/bin/kudu with pid 16790
I20260620 14:51:29.679816 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskUci3DU/build/asan/bin/kudu pid 16431
I20260620 14:51:29.680989 16494 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 14:51:29.795573 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskUci3DU/build/asan/bin/kudu with pid 16431
I20260620 14:51:49.817984 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskUci3DU/build/asan/bin/kudu
/tmp/dist-test-taskUci3DU/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskUci3DU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781966829091300-388-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskUci3DU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781966829091300-388-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskUci3DU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781966829091300-388-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskUci3DU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781966829091300-388-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:45085
--webserver_interface=127.0.97.62
--webserver_port=32859
--keytab_file=/tmp/dist-test-taskUci3DU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781966829091300-388-0/krb5kdc/kudu_127.0.97.62.keytab
--principal=kudu/127.0.97.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:44619
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.97.62:45085
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_rotation_seconds=20 with env {KRB5CCNAME=/tmp/dist-test-taskUci3DU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781966829091300-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskUci3DU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781966829091300-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskUci3DU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781966829091300-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 14:51:50.168076 16965 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 14:51:50.168406 16965 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 14:51:50.168565 16965 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 14:51:50.176296 16965 flags.cc:425] Enabled experimental flag: --authn_token_validity_seconds=20
W20260620 14:51:50.176409 16965 flags.cc:425] Enabled experimental flag: --authz_token_validity_seconds=20
W20260620 14:51:50.176491 16965 flags.cc:425] Enabled experimental flag: --tsk_rotation_seconds=20
W20260620 14:51:50.176568 16965 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 14:51:50.176664 16965 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 14:51:50.176792 16965 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 14:51:50.176874 16965 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 14:51:50.176960 16965 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 14:51:50.187801 16965 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:44619
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskUci3DU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781966829091300-388-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskUci3DU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781966829091300-388-0/minicluster-data/master-0/wal
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_rotation_seconds=20
--ipki_ca_key_size=768
--master_addresses=127.0.97.62:45085
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskUci3DU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781966829091300-388-0/krb5kdc/kudu_127.0.97.62.keytab
--principal=kudu/127.0.97.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:45085
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskUci3DU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781966829091300-388-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.97.62
--webserver_port=32859
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskUci3DU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781966829091300-388-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 13:41:20 UTC on ccycloud.quasar-vdagoi.root.comops.site
build id 4457462
ASAN enabled
I20260620 14:51:50.189185 16965 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 14:51:50.190606 16965 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 14:51:50.203459 16970 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 14:51:50.203361 16971 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 14:51:50.203441 16973 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 14:51:50.204111 16965 server_base.cc:1034] running on GCE node
I20260620 14:51:50.205070 16965 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 14:51:50.206697 16965 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 14:51:50.207968 16965 hybrid_clock.cc:648] HybridClock initialized: now 1781967110207890 us; error 52 us; skew 500 ppm
I20260620 14:51:50.208355 16965 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 14:51:50 dist-test-slave-q9j6 krb5kdc[15879](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781967110, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 14:51:50.212160 16965 init.cc:377] Logged in from keytab as kudu/127.0.97.62@KRBTEST.COM (short username kudu)
I20260620 14:51:50.213753 16965 webserver.cc:460] Webserver started at http://127.0.97.62:32859/ using document root <none> and password file <none>
I20260620 14:51:50.214550 16965 fs_manager.cc:359] Metadata directory not provided
I20260620 14:51:50.214718 16965 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 14:51:50.220870 16965 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.003s sys 0.001s
I20260620 14:51:50.224157 16980 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 14:51:50.225821 16965 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.002s
I20260620 14:51:50.225978 16965 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskUci3DU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781966829091300-388-0/minicluster-data/master-0/data,/tmp/dist-test-taskUci3DU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781966829091300-388-0/minicluster-data/master-0/wal
uuid: "0069e3cceda94cd68d41c5833a5174c9"
format_stamp: "Formatted at 2026-06-20 14:51:04 on dist-test-slave-q9j6"
server_key: "8a231203b28462a6eed146cdec780c4f"
server_key_iv: "1bb0c6765e03213cde26645f51bf2628"
server_key_version: "encryptionkey@0"
I20260620 14:51:50.226987 16965 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskUci3DU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781966829091300-388-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskUci3DU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781966829091300-388-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskUci3DU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781966829091300-388-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 14:51:50.242425 16965 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 14:51:50.247560 16965 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 14:51:50.247958 16965 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 14:51:50.266906 16965 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.62:45085
I20260620 14:51:50.266947 17031 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.62:45085 every 8 connection(s)
I20260620 14:51:50.268666 16965 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskUci3DU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781966829091300-388-0/minicluster-data/master-0/data/info.pb
I20260620 14:51:50.273200 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskUci3DU/build/asan/bin/kudu as pid 16965
I20260620 14:51:50.274061 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskUci3DU/build/asan/bin/kudu
/tmp/dist-test-taskUci3DU/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskUci3DU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781966829091300-388-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskUci3DU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781966829091300-388-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskUci3DU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781966829091300-388-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskUci3DU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781966829091300-388-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.1:40009
--local_ip_for_outbound_sockets=127.0.97.1
--tserver_master_addrs=127.0.97.62:45085
--webserver_port=34223
--webserver_interface=127.0.97.1
--keytab_file=/tmp/dist-test-taskUci3DU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781966829091300-388-0/krb5kdc/kudu_127.0.97.1.keytab
--principal=kudu/127.0.97.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:44619
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskUci3DU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781966829091300-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskUci3DU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781966829091300-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskUci3DU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781966829091300-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260620 14:51:50.276285 17032 sys_catalog.cc:263] Verifying existing consensus state
I20260620 14:51:50.280030 17032 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 0069e3cceda94cd68d41c5833a5174c9: Bootstrap starting.
I20260620 14:51:50.312497 17032 log.cc:824] T 00000000000000000000000000000000 P 0069e3cceda94cd68d41c5833a5174c9: Log is configured to *not* fsync() on all Append() calls
I20260620 14:51:50.335084 17032 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 0069e3cceda94cd68d41c5833a5174c9: Bootstrap replayed 1/1 log segments. Stats: ops{read=14 overwritten=0 applied=14 ignored=0} inserts{seen=8 ignored=0} mutations{seen=8 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260620 14:51:50.335896 17032 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 0069e3cceda94cd68d41c5833a5174c9: Bootstrap complete.
I20260620 14:51:50.351554 17032 raft_consensus.cc:348] T 00000000000000000000000000000000 P 0069e3cceda94cd68d41c5833a5174c9 [term 2 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0069e3cceda94cd68d41c5833a5174c9" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 45085 } }
I20260620 14:51:50.353000 17032 raft_consensus.cc:729] T 00000000000000000000000000000000 P 0069e3cceda94cd68d41c5833a5174c9 [term 2 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0069e3cceda94cd68d41c5833a5174c9, State: Initialized, Role: FOLLOWER
I20260620 14:51:50.353840 17032 consensus_queue.cc:260] T 00000000000000000000000000000000 P 0069e3cceda94cd68d41c5833a5174c9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 14, Last appended: 2.14, Last appended by leader: 14, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0069e3cceda94cd68d41c5833a5174c9" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 45085 } }
I20260620 14:51:50.354144 17032 raft_consensus.cc:388] T 00000000000000000000000000000000 P 0069e3cceda94cd68d41c5833a5174c9 [term 2 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 14:51:50.354281 17032 raft_consensus.cc:482] T 00000000000000000000000000000000 P 0069e3cceda94cd68d41c5833a5174c9 [term 2 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 14:51:50.354465 17032 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 0069e3cceda94cd68d41c5833a5174c9 [term 2 FOLLOWER]: Advancing to term 3
I20260620 14:51:50.360383 17032 raft_consensus.cc:504] T 00000000000000000000000000000000 P 0069e3cceda94cd68d41c5833a5174c9 [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0069e3cceda94cd68d41c5833a5174c9" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 45085 } }
I20260620 14:51:50.361073 17032 leader_election.cc:302] T 00000000000000000000000000000000 P 0069e3cceda94cd68d41c5833a5174c9 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 0069e3cceda94cd68d41c5833a5174c9; no voters:
I20260620 14:51:50.361805 17032 leader_election.cc:288] T 00000000000000000000000000000000 P 0069e3cceda94cd68d41c5833a5174c9 [CANDIDATE]: Term 3 election: Requested vote from peers
I20260620 14:51:50.361987 17036 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 0069e3cceda94cd68d41c5833a5174c9 [term 3 FOLLOWER]: Leader election won for term 3
I20260620 14:51:50.363229 17036 raft_consensus.cc:686] T 00000000000000000000000000000000 P 0069e3cceda94cd68d41c5833a5174c9 [term 3 LEADER]: Becoming Leader. State: Replica: 0069e3cceda94cd68d41c5833a5174c9, State: Running, Role: LEADER
I20260620 14:51:50.363914 17036 consensus_queue.cc:237] T 00000000000000000000000000000000 P 0069e3cceda94cd68d41c5833a5174c9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 14, Committed index: 14, Last appended: 2.14, Last appended by leader: 14, Current term: 3, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0069e3cceda94cd68d41c5833a5174c9" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 45085 } }
I20260620 14:51:50.364225 17032 sys_catalog.cc:564] T 00000000000000000000000000000000 P 0069e3cceda94cd68d41c5833a5174c9 [sys.catalog]: configured and running, proceeding with master startup.
I20260620 14:51:50.366622 17038 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0069e3cceda94cd68d41c5833a5174c9 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 0069e3cceda94cd68d41c5833a5174c9. Latest consensus state: current_term: 3 leader_uuid: "0069e3cceda94cd68d41c5833a5174c9" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0069e3cceda94cd68d41c5833a5174c9" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 45085 } } }
I20260620 14:51:50.367062 17038 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0069e3cceda94cd68d41c5833a5174c9 [sys.catalog]: This master's current role is: LEADER
I20260620 14:51:50.367691 17037 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0069e3cceda94cd68d41c5833a5174c9 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 3 leader_uuid: "0069e3cceda94cd68d41c5833a5174c9" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0069e3cceda94cd68d41c5833a5174c9" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 45085 } } }
I20260620 14:51:50.368455 17037 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0069e3cceda94cd68d41c5833a5174c9 [sys.catalog]: This master's current role is: LEADER
I20260620 14:51:50.377174 17041 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 14:51:50.390986 17041 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 14:51:50.392773 17041 catalog_manager.cc:1259] Loaded cluster ID: 85e8db040d0d4fe1baaac9f6b8446522
I20260620 14:51:50.392912 17041 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 14:51:50.397382 17041 catalog_manager.cc:1504] Loading token signing keys...
I20260620 14:51:50.398026 17054 catalog_manager.cc:795] Waiting for catalog manager background task thread to start: Service unavailable: Catalog manager is not initialized. State: Starting
I20260620 14:51:50.399425 17041 catalog_manager.cc:5879] T 00000000000000000000000000000000 P 0069e3cceda94cd68d41c5833a5174c9: Loaded TSK: 0
I20260620 14:51:50.399583 17041 catalog_manager.cc:5879] T 00000000000000000000000000000000 P 0069e3cceda94cd68d41c5833a5174c9: Loaded TSK: 1
F20260620 14:51:50.400800 17041 token_signing_key.cc:66] Check failed: _s.ok() Bad status: Runtime error: unable to load data from memory: error:0D0680A8:asn1 encoding routines:ASN1_CHECK_TLEN:wrong tag:tasn_dec.c:1220 error:0D06C03A:asn1 encoding routines:ASN1_D2I_EX_PRIMITIVE:nested asn1 error:tasn_dec.c:788 error:0D08303A:asn1 encoding routines:ASN1_TEMPLATE_NOEXP_D2I:nested asn1 error:tasn_dec.c:720:Field=version, Type=RSA error:04093004:rsa routines:OLD_RSA_PRIV_DECODE:RSA lib:rsa_ameth.c:121 error:0D0680A8:asn1 encoding routines:ASN1_CHECK_TLEN:wrong tag:tasn_dec.c:1220 error:0D06C03A:asn1 encoding routines:ASN1_D2I_EX_PRIMITIVE:nested asn1 error:tasn_dec.c:788 error:0D08303A:asn1 encoding routines:ASN1_TEMPLATE_NOEXP_D2I:nested asn1 error:tasn_dec.c:720:Field=version, Type=PKCS8_PRIV_KEY_INFO
*** Check failure stack trace: ***
@ 0x7f91f69a39ad google::LogMessage::Fail() at ??:0
@ 0x7f91f69a688a google::LogMessage::SendToLog() at ??:0
@ 0x7f91f69a34de google::LogMessage::Flush() at ??:0
@ 0x7f91f69a4a19 google::LogMessageFatal::~LogMessageFatal() at ??:0
@ 0x7f91fa7077d2 kudu::security::TokenSigningPrivateKey::TokenSigningPrivateKey() at ??:0
@ 0x7f91fa6f673e kudu::security::TokenSigner::ImportKeys() at ??:0
@ 0x7f920c3d21c0 kudu::master::CatalogManager::LoadTskEntries() at ??:0
@ 0x7f920c3d1783 kudu::master::CatalogManager::InitTokenSigner() at ??:0
@ 0x7f920c45d2a7 kudu::master::CatalogManager::PrepareForLeadershipTask()::$_10::operator()() at ??:0
@ 0x7f920c45d0c4 std::_Function_handler<>::_M_invoke() at ??:0
@ 0x7f920c4a75e9 std::function<>::operator()() at ??:0
@ 0x7f920c3d5cdd kudu::master::CatalogManager::PrepareForLeadershipTask()::$_6::operator()() at ??:0
@ 0x7f920c3d3c36 kudu::master::CatalogManager::PrepareForLeadershipTask() at ??:0
@ 0x7f920c45b610 kudu::master::CatalogManager::ElectedAsLeaderCb()::$_2::operator()() at ??:0
@ 0x7f920c45b439 std::_Function_handler<>::_M_invoke() at ??:0
@ 0x9edb7d std::function<>::operator()() at ??:?
@ 0x7f91f95ef345 kudu::ThreadPool::DispatchThread() at ??:0
@ 0x7f91f95f291c kudu::ThreadPool::CreateThread()::$_2::operator()() at ??:0
@ 0x7f91f95f2799 std::_Function_handler<>::_M_invoke() at ??:0
@ 0x9edb7d std::function<>::operator()() at ??:?
@ 0x7f91f95cdeaf kudu::Thread::SuperviseThread() at ??:0
@ 0x7f91fc8a76db start_thread at ??:0
@ 0x7f91f583d61f clone at ??:0
W20260620 14:51:50.748148 17034 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 14:51:50.748446 17034 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 14:51:50.748543 17034 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 14:51:50.756240 17034 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 14:51:50.756366 17034 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 14:51:50.756482 17034 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.1
I20260620 14:51:50.766726 17034 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:44619
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskUci3DU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781966829091300-388-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskUci3DU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781966829091300-388-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskUci3DU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781966829091300-388-0/krb5kdc/kudu_127.0.97.1.keytab
--principal=kudu/127.0.97.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.1:40009
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskUci3DU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781966829091300-388-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.97.1
--webserver_port=34223
--tserver_master_addrs=127.0.97.62:45085
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.1
--log_dir=/tmp/dist-test-taskUci3DU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781966829091300-388-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 13:41:20 UTC on ccycloud.quasar-vdagoi.root.comops.site
build id 4457462
ASAN enabled
I20260620 14:51:50.768146 17034 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 14:51:50.769457 17034 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 14:51:50.783704 17081 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 14:51:50.783644 17083 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 14:51:50.783740 17080 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 14:51:50.784523 17034 server_base.cc:1034] running on GCE node
I20260620 14:51:50.785193 17034 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 14:51:50.786201 17034 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 14:51:50.787458 17034 hybrid_clock.cc:648] HybridClock initialized: now 1781967110787396 us; error 52 us; skew 500 ppm
I20260620 14:51:50.788033 17034 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 14:51:50 dist-test-slave-q9j6 krb5kdc[15879](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781967110, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 14:51:50.791296 17034 init.cc:377] Logged in from keytab as kudu/127.0.97.1@KRBTEST.COM (short username kudu)
I20260620 14:51:50.792914 17034 webserver.cc:460] Webserver started at http://127.0.97.1:34223/ using document root <none> and password file <none>
I20260620 14:51:50.793581 17034 fs_manager.cc:359] Metadata directory not provided
I20260620 14:51:50.793702 17034 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 14:51:50.799893 17034 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.003s sys 0.001s
I20260620 14:51:50.802666 17090 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 14:51:50.804178 17034 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260620 14:51:50.804315 17034 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskUci3DU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781966829091300-388-0/minicluster-data/ts-0/data,/tmp/dist-test-taskUci3DU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781966829091300-388-0/minicluster-data/ts-0/wal
uuid: "531130dea5ed49f5926726011f1dc612"
format_stamp: "Formatted at 2026-06-20 14:51:04 on dist-test-slave-q9j6"
server_key: "5e2e2b609987b64726862da0486862ef"
server_key_iv: "8a2d2e54a4b49939c308f7beab05f356"
server_key_version: "encryptionkey@0"
I20260620 14:51:50.805269 17034 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskUci3DU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781966829091300-388-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskUci3DU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781966829091300-388-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskUci3DU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781966829091300-388-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 14:51:50.823827 17034 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 14:51:50.828334 17034 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 14:51:50.828599 17034 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 14:51:50.829574 17034 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 14:51:50.830899 17034 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 14:51:50.831007 17034 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 14:51:50.831094 17034 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 14:51:50.831148 17034 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 14:51:50.866577 17034 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.1:40009
I20260620 14:51:50.866637 17202 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.1:40009 every 8 connection(s)
I20260620 14:51:50.868234 17034 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskUci3DU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781966829091300-388-0/minicluster-data/ts-0/data/info.pb
I20260620 14:51:50.873479 17205 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:51:50.871448 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:45085 (local address 127.0.97.1:37505)
0620 14:51:50.872289 (+ 841us) negotiation.cc:107] Waiting for socket to connect
0620 14:51:50.872497 (+ 208us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:45085: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":559,"thread_start_us":368,"threads_started":1}
W20260620 14:51:50.874292 17203 heartbeater.cc:644] Failed to heartbeat to 127.0.97.62:45085 (0 consecutive failures): Network error: Failed to ping master at 127.0.97.62:45085: Client connection negotiation failed: client connection to 127.0.97.62:45085: connect: Connection refused (error 111)
I20260620 14:51:50.875150 17205 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:51:50.874823 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:45085 (local address 127.0.97.1:39027)
0620 14:51:50.874967 (+ 144us) negotiation.cc:107] Waiting for socket to connect
0620 14:51:50.875040 (+ 73us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:45085: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":50}
I20260620 14:51:50.875670 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskUci3DU/build/asan/bin/kudu as pid 17034
I20260620 14:51:50.876138 17205 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:51:50.875836 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:45085 (local address 127.0.97.1:56093)
0620 14:51:50.875949 (+ 113us) negotiation.cc:107] Waiting for socket to connect
0620 14:51:50.876022 (+ 73us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:45085: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":33}
W20260620 14:51:50.876408 17203 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
I20260620 14:51:50.876453 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskUci3DU/build/asan/bin/kudu
/tmp/dist-test-taskUci3DU/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskUci3DU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781966829091300-388-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskUci3DU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781966829091300-388-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskUci3DU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781966829091300-388-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskUci3DU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781966829091300-388-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.2:32783
--local_ip_for_outbound_sockets=127.0.97.2
--tserver_master_addrs=127.0.97.62:45085
--webserver_port=40609
--webserver_interface=127.0.97.2
--keytab_file=/tmp/dist-test-taskUci3DU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781966829091300-388-0/krb5kdc/kudu_127.0.97.2.keytab
--principal=kudu/127.0.97.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:44619
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskUci3DU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781966829091300-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskUci3DU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781966829091300-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskUci3DU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781966829091300-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 14:51:51.234990 17206 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 14:51:51.235292 17206 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 14:51:51.235399 17206 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 14:51:51.242998 17206 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 14:51:51.243124 17206 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 14:51:51.243242 17206 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.2
I20260620 14:51:51.253499 17206 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:44619
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskUci3DU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781966829091300-388-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskUci3DU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781966829091300-388-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskUci3DU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781966829091300-388-0/krb5kdc/kudu_127.0.97.2.keytab
--principal=kudu/127.0.97.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.2:32783
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskUci3DU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781966829091300-388-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.97.2
--webserver_port=40609
--tserver_master_addrs=127.0.97.62:45085
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.2
--log_dir=/tmp/dist-test-taskUci3DU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781966829091300-388-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 13:41:20 UTC on ccycloud.quasar-vdagoi.root.comops.site
build id 4457462
ASAN enabled
I20260620 14:51:51.254814 17206 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 14:51:51.256146 17206 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 14:51:51.270040 17214 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 14:51:51.270116 17212 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 14:51:51.270113 17211 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 14:51:51.271034 17206 server_base.cc:1034] running on GCE node
I20260620 14:51:51.271858 17206 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 14:51:51.272934 17206 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 14:51:51.274183 17206 hybrid_clock.cc:648] HybridClock initialized: now 1781967111274138 us; error 46 us; skew 500 ppm
I20260620 14:51:51.274567 17206 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 14:51:51 dist-test-slave-q9j6 krb5kdc[15879](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781967111, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 14:51:51.277886 17206 init.cc:377] Logged in from keytab as kudu/127.0.97.2@KRBTEST.COM (short username kudu)
I20260620 14:51:51.279521 17206 webserver.cc:460] Webserver started at http://127.0.97.2:40609/ using document root <none> and password file <none>
I20260620 14:51:51.280354 17206 fs_manager.cc:359] Metadata directory not provided
I20260620 14:51:51.280506 17206 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 14:51:51.286321 17206 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.005s sys 0.001s
I20260620 14:51:51.289314 17221 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 14:51:51.290795 17206 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260620 14:51:51.290992 17206 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskUci3DU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781966829091300-388-0/minicluster-data/ts-1/data,/tmp/dist-test-taskUci3DU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781966829091300-388-0/minicluster-data/ts-1/wal
uuid: "ff16403da3304c18aff565fade86842e"
format_stamp: "Formatted at 2026-06-20 14:51:05 on dist-test-slave-q9j6"
server_key: "028d91d47f18dd8924ae2d375a9553de"
server_key_iv: "f326979f277e705d2dc12dfd1218c78c"
server_key_version: "encryptionkey@0"
I20260620 14:51:51.292128 17206 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskUci3DU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781966829091300-388-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskUci3DU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781966829091300-388-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskUci3DU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781966829091300-388-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 14:51:51.310547 17206 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 14:51:51.315701 17206 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 14:51:51.316020 17206 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 14:51:51.317119 17206 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 14:51:51.318580 17206 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 14:51:51.318719 17206 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 14:51:51.318887 17206 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 14:51:51.318984 17206 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 14:51:51.354151 17206 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.2:32783
I20260620 14:51:51.354249 17333 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.2:32783 every 8 connection(s)
I20260620 14:51:51.355906 17206 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskUci3DU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781966829091300-388-0/minicluster-data/ts-1/data/info.pb
I20260620 14:51:51.361269 17336 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:51:51.359185 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:45085 (local address 127.0.97.2:52545)
0620 14:51:51.360069 (+ 884us) negotiation.cc:107] Waiting for socket to connect
0620 14:51:51.360282 (+ 213us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:45085: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":603,"thread_start_us":385,"threads_started":1}
W20260620 14:51:51.362141 17334 heartbeater.cc:644] Failed to heartbeat to 127.0.97.62:45085 (0 consecutive failures): Network error: Failed to ping master at 127.0.97.62:45085: Client connection negotiation failed: client connection to 127.0.97.62:45085: connect: Connection refused (error 111)
I20260620 14:51:51.363205 17336 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:51:51.362826 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:45085 (local address 127.0.97.2:35673)
0620 14:51:51.362987 (+ 161us) negotiation.cc:107] Waiting for socket to connect
0620 14:51:51.363063 (+ 76us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:45085: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":60}
I20260620 14:51:51.363494 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskUci3DU/build/asan/bin/kudu as pid 17206
I20260620 14:51:51.364284 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskUci3DU/build/asan/bin/kudu
/tmp/dist-test-taskUci3DU/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskUci3DU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781966829091300-388-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskUci3DU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781966829091300-388-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskUci3DU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781966829091300-388-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskUci3DU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781966829091300-388-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.3:33429
--local_ip_for_outbound_sockets=127.0.97.3
--tserver_master_addrs=127.0.97.62:45085
--webserver_port=36959
--webserver_interface=127.0.97.3
--keytab_file=/tmp/dist-test-taskUci3DU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781966829091300-388-0/krb5kdc/kudu_127.0.97.3.keytab
--principal=kudu/127.0.97.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:44619
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskUci3DU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781966829091300-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskUci3DU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781966829091300-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskUci3DU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781966829091300-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260620 14:51:51.364336 17336 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:51:51.363880 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:45085 (local address 127.0.97.2:47069)
0620 14:51:51.364057 (+ 177us) negotiation.cc:107] Waiting for socket to connect
0620 14:51:51.364167 (+ 110us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:45085: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":61}
W20260620 14:51:51.364674 17334 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260620 14:51:51.705245 17337 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 14:51:51.705560 17337 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 14:51:51.705745 17337 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 14:51:51.713483 17337 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 14:51:51.713624 17337 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 14:51:51.713831 17337 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.3
I20260620 14:51:51.724327 17337 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:44619
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskUci3DU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781966829091300-388-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskUci3DU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781966829091300-388-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskUci3DU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781966829091300-388-0/krb5kdc/kudu_127.0.97.3.keytab
--principal=kudu/127.0.97.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.3:33429
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskUci3DU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781966829091300-388-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.97.3
--webserver_port=36959
--tserver_master_addrs=127.0.97.62:45085
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.3
--log_dir=/tmp/dist-test-taskUci3DU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781966829091300-388-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 13:41:20 UTC on ccycloud.quasar-vdagoi.root.comops.site
build id 4457462
ASAN enabled
I20260620 14:51:51.725747 17337 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 14:51:51.727164 17337 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 14:51:51.741004 17345 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 14:51:51.741055 17342 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 14:51:51.741076 17343 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 14:51:51.742566 17337 server_base.cc:1034] running on GCE node
I20260620 14:51:51.743183 17337 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 14:51:51.744313 17337 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 14:51:51.745532 17337 hybrid_clock.cc:648] HybridClock initialized: now 1781967111745484 us; error 34 us; skew 500 ppm
I20260620 14:51:51.745922 17337 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 14:51:51 dist-test-slave-q9j6 krb5kdc[15879](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781967111, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 14:51:51.749277 17337 init.cc:377] Logged in from keytab as kudu/127.0.97.3@KRBTEST.COM (short username kudu)
I20260620 14:51:51.750891 17337 webserver.cc:460] Webserver started at http://127.0.97.3:36959/ using document root <none> and password file <none>
I20260620 14:51:51.751649 17337 fs_manager.cc:359] Metadata directory not provided
I20260620 14:51:51.751776 17337 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 14:51:51.757557 17337 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.004s sys 0.001s
I20260620 14:51:51.760537 17352 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 14:51:51.762072 17337 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.001s
I20260620 14:51:51.762264 17337 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskUci3DU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781966829091300-388-0/minicluster-data/ts-2/data,/tmp/dist-test-taskUci3DU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781966829091300-388-0/minicluster-data/ts-2/wal
uuid: "6879ba8da72645c29edc506ff924ba31"
format_stamp: "Formatted at 2026-06-20 14:51:05 on dist-test-slave-q9j6"
server_key: "d36a4e4bd2f0abafa50e8a532e9e4c45"
server_key_iv: "bac9ca16940dc3b8aa0a4af457f0ce41"
server_key_version: "encryptionkey@0"
I20260620 14:51:51.763348 17337 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskUci3DU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781966829091300-388-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskUci3DU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781966829091300-388-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskUci3DU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781966829091300-388-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 14:51:51.776952 17337 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 14:51:51.781489 17337 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 14:51:51.781785 17337 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 14:51:51.782797 17337 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 14:51:51.784271 17337 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 14:51:51.784359 17337 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 14:51:51.784519 17337 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 14:51:51.784622 17337 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 14:51:51.819887 17337 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.3:33429
I20260620 14:51:51.819977 17464 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.3:33429 every 8 connection(s)
I20260620 14:51:51.821517 17337 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskUci3DU/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781966829091300-388-0/minicluster-data/ts-2/data/info.pb
I20260620 14:51:51.826743 17467 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:51:51.824785 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:45085 (local address 127.0.97.3:41399)
0620 14:51:51.825558 (+ 773us) negotiation.cc:107] Waiting for socket to connect
0620 14:51:51.825771 (+ 213us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:45085: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":511,"thread_start_us":304,"threads_started":1}
W20260620 14:51:51.827450 17465 heartbeater.cc:644] Failed to heartbeat to 127.0.97.62:45085 (0 consecutive failures): Network error: Failed to ping master at 127.0.97.62:45085: Client connection negotiation failed: client connection to 127.0.97.62:45085: connect: Connection refused (error 111)
I20260620 14:51:51.828625 17467 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:51:51.828113 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:45085 (local address 127.0.97.3:53133)
0620 14:51:51.828299 (+ 186us) negotiation.cc:107] Waiting for socket to connect
0620 14:51:51.828391 (+ 92us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:45085: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":45}
I20260620 14:51:51.829578 17467 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:51:51.829246 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:45085 (local address 127.0.97.3:40037)
0620 14:51:51.829385 (+ 139us) negotiation.cc:107] Waiting for socket to connect
0620 14:51:51.829458 (+ 73us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:45085: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":60}
I20260620 14:51:51.829797 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskUci3DU/build/asan/bin/kudu as pid 17337
W20260620 14:51:51.829918 17465 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
I20260620 14:51:51.877835 17469 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:51:51.877015 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:45085 (local address 127.0.97.1:49101)
0620 14:51:51.877493 (+ 478us) negotiation.cc:107] Waiting for socket to connect
0620 14:51:51.877593 (+ 100us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:45085: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":367,"thread_start_us":236,"threads_started":1}
I20260620 14:51:52.366173 17470 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:51:52.365330 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:45085 (local address 127.0.97.2:38907)
0620 14:51:52.365843 (+ 513us) negotiation.cc:107] Waiting for socket to connect
0620 14:51:52.365936 (+ 93us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:45085: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":401,"thread_start_us":265,"threads_started":1}
I20260620 14:51:52.831351 17471 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:51:52.830482 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:45085 (local address 127.0.97.3:41467)
0620 14:51:52.831011 (+ 529us) negotiation.cc:107] Waiting for socket to connect
0620 14:51:52.831107 (+ 96us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:45085: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":404,"thread_start_us":262,"threads_started":1}
I20260620 14:51:52.879734 17472 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:51:52.878884 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:45085 (local address 127.0.97.1:35095)
0620 14:51:52.879384 (+ 500us) negotiation.cc:107] Waiting for socket to connect
0620 14:51:52.879481 (+ 97us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:45085: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":393,"thread_start_us":252,"threads_started":1}
I20260620 14:51:53.368233 17473 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:51:53.367396 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:45085 (local address 127.0.97.2:57499)
0620 14:51:53.367964 (+ 568us) negotiation.cc:107] Waiting for socket to connect
0620 14:51:53.368063 (+ 99us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:45085: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":455,"thread_start_us":288,"threads_started":1}
I20260620 14:51:53.833196 17474 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:51:53.832408 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:45085 (local address 127.0.97.3:35509)
0620 14:51:53.832930 (+ 522us) negotiation.cc:107] Waiting for socket to connect
0620 14:51:53.833028 (+ 98us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:45085: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":414,"thread_start_us":238,"threads_started":1}
I20260620 14:51:53.881330 17475 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:51:53.880634 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:45085 (local address 127.0.97.1:53969)
0620 14:51:53.881066 (+ 432us) negotiation.cc:107] Waiting for socket to connect
0620 14:51:53.881153 (+ 87us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:45085: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":331,"thread_start_us":224,"threads_started":1}
I20260620 14:51:54.369992 17476 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:51:54.369187 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:45085 (local address 127.0.97.2:40089)
0620 14:51:54.369696 (+ 509us) negotiation.cc:107] Waiting for socket to connect
0620 14:51:54.369789 (+ 93us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:45085: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":400,"thread_start_us":273,"threads_started":1}
I20260620 14:51:54.835040 17477 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:51:54.834237 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:45085 (local address 127.0.97.3:33469)
0620 14:51:54.834753 (+ 516us) negotiation.cc:107] Waiting for socket to connect
0620 14:51:54.834845 (+ 92us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:45085: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":401,"thread_start_us":268,"threads_started":1}
I20260620 14:51:54.883220 17478 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:51:54.882461 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:45085 (local address 127.0.97.1:55685)
0620 14:51:54.882926 (+ 465us) negotiation.cc:107] Waiting for socket to connect
0620 14:51:54.883015 (+ 89us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:45085: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":351,"thread_start_us":224,"threads_started":1}
I20260620 14:51:55.372056 17479 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:51:55.371159 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:45085 (local address 127.0.97.2:54379)
0620 14:51:55.371766 (+ 607us) negotiation.cc:107] Waiting for socket to connect
0620 14:51:55.371856 (+ 90us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:45085: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":459,"thread_start_us":257,"threads_started":1}
I20260620 14:51:55.836799 17480 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:51:55.835991 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:45085 (local address 127.0.97.3:33465)
0620 14:51:55.836481 (+ 490us) negotiation.cc:107] Waiting for socket to connect
0620 14:51:55.836590 (+ 109us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:45085: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":378,"thread_start_us":243,"threads_started":1}
I20260620 14:51:55.885102 17481 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:51:55.884351 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:45085 (local address 127.0.97.1:43545)
0620 14:51:55.884813 (+ 462us) negotiation.cc:107] Waiting for socket to connect
0620 14:51:55.884896 (+ 83us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:45085: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":359,"thread_start_us":229,"threads_started":1}
I20260620 14:51:56.374022 17482 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:51:56.373170 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:45085 (local address 127.0.97.2:41953)
0620 14:51:56.373677 (+ 507us) negotiation.cc:107] Waiting for socket to connect
0620 14:51:56.373771 (+ 94us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:45085: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":396,"thread_start_us":254,"threads_started":1}
I20260620 14:51:56.838670 17483 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:51:56.837900 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:45085 (local address 127.0.97.3:52547)
0620 14:51:56.838406 (+ 506us) negotiation.cc:107] Waiting for socket to connect
0620 14:51:56.838498 (+ 92us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:45085: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":390,"thread_start_us":257,"threads_started":1}
I20260620 14:51:56.886912 17484 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:51:56.886099 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:45085 (local address 127.0.97.1:42651)
0620 14:51:56.886641 (+ 542us) negotiation.cc:107] Waiting for socket to connect
0620 14:51:56.886742 (+ 101us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:45085: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":434,"thread_start_us":289,"threads_started":1}
I20260620 14:51:57.375965 17485 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:51:57.375121 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:45085 (local address 127.0.97.2:44545)
0620 14:51:57.375670 (+ 549us) negotiation.cc:107] Waiting for socket to connect
0620 14:51:57.375780 (+ 110us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:45085: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":377,"thread_start_us":246,"threads_started":1}
I20260620 14:51:57.840780 17486 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:51:57.839948 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:45085 (local address 127.0.97.3:57287)
0620 14:51:57.840504 (+ 556us) negotiation.cc:107] Waiting for socket to connect
0620 14:51:57.840594 (+ 90us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:45085: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":437,"thread_start_us":309,"threads_started":1}
I20260620 14:51:57.888837 17487 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:51:57.887966 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:45085 (local address 127.0.97.1:56703)
0620 14:51:57.888511 (+ 545us) negotiation.cc:107] Waiting for socket to connect
0620 14:51:57.888603 (+ 92us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:45085: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":421,"thread_start_us":235,"threads_started":1}
I20260620 14:51:58.378031 17488 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:51:58.377109 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:45085 (local address 127.0.97.2:42831)
0620 14:51:58.377685 (+ 576us) negotiation.cc:107] Waiting for socket to connect
0620 14:51:58.377781 (+ 96us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:45085: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":449,"thread_start_us":252,"threads_started":1}
I20260620 14:51:58.842794 17489 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:51:58.841914 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:45085 (local address 127.0.97.3:38753)
0620 14:51:58.842472 (+ 558us) negotiation.cc:107] Waiting for socket to connect
0620 14:51:58.842567 (+ 95us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:45085: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":447,"thread_start_us":258,"threads_started":1}
I20260620 14:51:58.890717 17490 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:51:58.889950 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:45085 (local address 127.0.97.1:48467)
0620 14:51:58.890402 (+ 452us) negotiation.cc:107] Waiting for socket to connect
0620 14:51:58.890490 (+ 88us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:45085: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":344,"thread_start_us":224,"threads_started":1}
I20260620 14:51:59.379858 17491 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:51:59.379014 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:45085 (local address 127.0.97.2:49291)
0620 14:51:59.379500 (+ 486us) negotiation.cc:107] Waiting for socket to connect
0620 14:51:59.379587 (+ 87us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:45085: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":382,"thread_start_us":251,"threads_started":1}
I20260620 14:51:59.844573 17492 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:51:59.843811 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:45085 (local address 127.0.97.3:34229)
0620 14:51:59.844304 (+ 493us) negotiation.cc:107] Waiting for socket to connect
0620 14:51:59.844396 (+ 92us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:45085: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":375,"thread_start_us":237,"threads_started":1}
I20260620 14:51:59.892639 17493 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:51:59.891833 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:45085 (local address 127.0.97.1:46603)
0620 14:51:59.892361 (+ 528us) negotiation.cc:107] Waiting for socket to connect
0620 14:51:59.892456 (+ 95us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:45085: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":403,"thread_start_us":240,"threads_started":1}
I20260620 14:52:00.381737 17494 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:52:00.380925 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:45085 (local address 127.0.97.2:39209)
0620 14:52:00.381463 (+ 538us) negotiation.cc:107] Waiting for socket to connect
0620 14:52:00.381555 (+ 92us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:45085: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":423,"thread_start_us":272,"threads_started":1}
I20260620 14:52:00.846815 17495 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:52:00.845632 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:45085 (local address 127.0.97.3:33663)
0620 14:52:00.846303 (+ 671us) negotiation.cc:107] Waiting for socket to connect
0620 14:52:00.846549 (+ 246us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:45085: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":541,"thread_start_us":343,"threads_started":1}
I20260620 14:52:00.894340 17496 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:52:00.893602 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:45085 (local address 127.0.97.1:59731)
0620 14:52:00.894076 (+ 474us) negotiation.cc:107] Waiting for socket to connect
0620 14:52:00.894163 (+ 87us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:45085: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":368,"thread_start_us":255,"threads_started":1}
I20260620 14:52:01.383764 17497 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:52:01.382911 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:45085 (local address 127.0.97.2:54753)
0620 14:52:01.383414 (+ 503us) negotiation.cc:107] Waiting for socket to connect
0620 14:52:01.383513 (+ 99us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:45085: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":395,"thread_start_us":277,"threads_started":1}
I20260620 14:52:01.849133 17498 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:52:01.848278 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:45085 (local address 127.0.97.3:39383)
0620 14:52:01.848873 (+ 595us) negotiation.cc:107] Waiting for socket to connect
0620 14:52:01.848964 (+ 91us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:45085: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":475,"thread_start_us":339,"threads_started":1}
I20260620 14:52:01.896189 17499 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:52:01.895300 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:45085 (local address 127.0.97.1:43829)
0620 14:52:01.895842 (+ 542us) negotiation.cc:107] Waiting for socket to connect
0620 14:52:01.895935 (+ 93us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:45085: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":436,"thread_start_us":313,"threads_started":1}
I20260620 14:52:02.385597 17500 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:52:02.384826 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:45085 (local address 127.0.97.2:34727)
0620 14:52:02.385302 (+ 476us) negotiation.cc:107] Waiting for socket to connect
0620 14:52:02.385407 (+ 105us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:45085: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":368,"thread_start_us":232,"threads_started":1}
I20260620 14:52:02.850929 17501 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:52:02.850174 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:45085 (local address 127.0.97.3:42595)
0620 14:52:02.850657 (+ 483us) negotiation.cc:107] Waiting for socket to connect
0620 14:52:02.850747 (+ 90us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:45085: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":364,"thread_start_us":241,"threads_started":1}
I20260620 14:52:02.897921 17502 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:52:02.897227 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:45085 (local address 127.0.97.1:47269)
0620 14:52:02.897674 (+ 447us) negotiation.cc:107] Waiting for socket to connect
0620 14:52:02.897763 (+ 89us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:45085: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":332,"thread_start_us":219,"threads_started":1}
I20260620 14:52:03.387490 17503 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:52:03.386695 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:45085 (local address 127.0.97.2:53845)
0620 14:52:03.387222 (+ 527us) negotiation.cc:107] Waiting for socket to connect
0620 14:52:03.387319 (+ 97us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:45085: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":393,"thread_start_us":264,"threads_started":1}
I20260620 14:52:03.853060 17504 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:52:03.852176 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:45085 (local address 127.0.97.3:59605)
0620 14:52:03.852730 (+ 554us) negotiation.cc:107] Waiting for socket to connect
0620 14:52:03.852814 (+ 84us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:45085: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":418,"thread_start_us":307,"threads_started":1}
I20260620 14:52:03.899703 17505 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:52:03.898810 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:45085 (local address 127.0.97.1:44047)
0620 14:52:03.899306 (+ 496us) negotiation.cc:107] Waiting for socket to connect
0620 14:52:03.899413 (+ 107us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:45085: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":387,"thread_start_us":263,"threads_started":1}
I20260620 14:52:04.389364 17506 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:52:04.388542 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:45085 (local address 127.0.97.2:44445)
0620 14:52:04.389065 (+ 523us) negotiation.cc:107] Waiting for socket to connect
0620 14:52:04.389161 (+ 96us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:45085: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":415,"thread_start_us":257,"threads_started":1}
I20260620 14:52:04.855346 17507 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:52:04.854428 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:45085 (local address 127.0.97.3:54067)
0620 14:52:04.855062 (+ 634us) negotiation.cc:107] Waiting for socket to connect
0620 14:52:04.855157 (+ 95us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:45085: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":492,"thread_start_us":321,"threads_started":1}
I20260620 14:52:04.901472 17508 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:52:04.900707 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:45085 (local address 127.0.97.1:35795)
0620 14:52:04.901190 (+ 483us) negotiation.cc:107] Waiting for socket to connect
0620 14:52:04.901286 (+ 96us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:45085: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":373,"thread_start_us":254,"threads_started":1}
I20260620 14:52:05.391453 17509 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:52:05.390446 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:45085 (local address 127.0.97.2:54583)
0620 14:52:05.391068 (+ 622us) negotiation.cc:107] Waiting for socket to connect
0620 14:52:05.391195 (+ 127us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:45085: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":482,"thread_start_us":317,"threads_started":1}
I20260620 14:52:05.857141 17510 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:52:05.856315 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:45085 (local address 127.0.97.3:51207)
0620 14:52:05.856775 (+ 460us) negotiation.cc:107] Waiting for socket to connect
0620 14:52:05.856905 (+ 130us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:45085: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":355,"thread_start_us":228,"threads_started":1}
I20260620 14:52:05.903435 17511 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:52:05.902682 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:45085 (local address 127.0.97.1:40141)
0620 14:52:05.903151 (+ 469us) negotiation.cc:107] Waiting for socket to connect
0620 14:52:05.903245 (+ 94us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:45085: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":352,"thread_start_us":229,"threads_started":1}
I20260620 14:52:06.393468 17512 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:52:06.392712 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:45085 (local address 127.0.97.2:39839)
0620 14:52:06.393179 (+ 467us) negotiation.cc:107] Waiting for socket to connect
0620 14:52:06.393278 (+ 99us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:45085: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":353,"thread_start_us":234,"threads_started":1}
I20260620 14:52:06.837848 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskUci3DU/build/asan/bin/kudu pid 17034
I20260620 14:52:06.859102 17514 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:52:06.858370 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:45085 (local address 127.0.97.3:57565)
0620 14:52:06.858834 (+ 464us) negotiation.cc:107] Waiting for socket to connect
0620 14:52:06.858936 (+ 102us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:45085: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":349,"thread_start_us":247,"threads_started":1}
I20260620 14:52:06.860921 17513 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:52:06.839194 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46298 (local address 127.0.97.1:40009)
0620 14:52:06.839988 (+ 794us) server_negotiation.cc:206] Beginning negotiation
0620 14:52:06.840023 (+ 35us) server_negotiation.cc:399] Waiting for connection header
0620 14:52:06.840076 (+ 53us) server_negotiation.cc:407] Connection header received
0620 14:52:06.841887 (+ 1811us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:52:06.841898 (+ 11us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:52:06.842373 (+ 475us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:52:06.842690 (+ 317us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:52:06.844234 (+ 1544us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:52:06.845539 (+ 1305us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:52:06.846790 (+ 1251us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:52:06.847519 (+ 729us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:52:06.847760 (+ 241us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:52:06.854008 (+ 6248us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 14:52:06.854024 (+ 16us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 14:52:06.854045 (+ 21us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 14:52:06.854099 (+ 54us) server_negotiation.cc:922] calling sasl_server_start()
0620 14:52:06.857070 (+ 2971us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:52:06.857934 (+ 864us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:52:06.857942 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:52:06.857949 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:52:06.858200 (+ 251us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:52:06.858603 (+ 403us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:52:06.858611 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:52:06.858617 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:52:06.859339 (+ 722us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 14:52:06.859713 (+ 374us) server_negotiation.cc:1035] Waiting for connection context
0620 14:52:06.860165 (+ 452us) server_negotiation.cc:299] Negotiation successful
0620 14:52:06.860635 (+ 470us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":494,"thread_start_us":318,"threads_started":1}
I20260620 14:52:06.862370 17198 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 14:52:06.948705 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskUci3DU/build/asan/bin/kudu with pid 17034
I20260620 14:52:06.966845 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskUci3DU/build/asan/bin/kudu pid 17206
I20260620 14:52:06.985435 17516 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:52:06.967943 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41150 (local address 127.0.97.2:32783)
0620 14:52:06.968633 (+ 690us) server_negotiation.cc:206] Beginning negotiation
0620 14:52:06.968647 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0620 14:52:06.968685 (+ 38us) server_negotiation.cc:407] Connection header received
0620 14:52:06.970262 (+ 1577us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:52:06.970271 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:52:06.970743 (+ 472us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:52:06.971107 (+ 364us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:52:06.972653 (+ 1546us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:52:06.973889 (+ 1236us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:52:06.975172 (+ 1283us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:52:06.975947 (+ 775us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:52:06.976148 (+ 201us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:52:06.978756 (+ 2608us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 14:52:06.978764 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 14:52:06.978782 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 14:52:06.978830 (+ 48us) server_negotiation.cc:922] calling sasl_server_start()
0620 14:52:06.981701 (+ 2871us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:52:06.982450 (+ 749us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:52:06.982459 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:52:06.982467 (+ 8us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:52:06.982651 (+ 184us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:52:06.983054 (+ 403us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:52:06.983060 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:52:06.983063 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:52:06.983758 (+ 695us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 14:52:06.984110 (+ 352us) server_negotiation.cc:1035] Waiting for connection context
0620 14:52:06.984681 (+ 571us) server_negotiation.cc:299] Negotiation successful
0620 14:52:06.985144 (+ 463us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":487,"thread_start_us":345,"threads_started":1}
I20260620 14:52:06.986889 17329 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 14:52:07.078181 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskUci3DU/build/asan/bin/kudu with pid 17206
I20260620 14:52:07.099709 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskUci3DU/build/asan/bin/kudu pid 17337
I20260620 14:52:07.123835 17518 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:52:07.100765 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33152 (local address 127.0.97.3:33429)
0620 14:52:07.101367 (+ 602us) server_negotiation.cc:206] Beginning negotiation
0620 14:52:07.101387 (+ 20us) server_negotiation.cc:399] Waiting for connection header
0620 14:52:07.101441 (+ 54us) server_negotiation.cc:407] Connection header received
0620 14:52:07.104052 (+ 2611us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:52:07.104062 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:52:07.104536 (+ 474us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:52:07.104984 (+ 448us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:52:07.107882 (+ 2898us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:52:07.109958 (+ 2076us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:52:07.111880 (+ 1922us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:52:07.113104 (+ 1224us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:52:07.113352 (+ 248us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:52:07.116267 (+ 2915us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 14:52:07.116279 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 14:52:07.116301 (+ 22us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 14:52:07.116383 (+ 82us) server_negotiation.cc:922] calling sasl_server_start()
0620 14:52:07.120252 (+ 3869us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:52:07.121032 (+ 780us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:52:07.121042 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:52:07.121052 (+ 10us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:52:07.121237 (+ 185us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:52:07.121632 (+ 395us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:52:07.121637 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:52:07.121641 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:52:07.122225 (+ 584us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 14:52:07.122552 (+ 327us) server_negotiation.cc:1035] Waiting for connection context
0620 14:52:07.122998 (+ 446us) server_negotiation.cc:299] Negotiation successful
0620 14:52:07.123445 (+ 447us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":394,"thread_start_us":592,"threads_started":1}
I20260620 14:52:07.125801 17460 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 14:52:07.213385 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskUci3DU/build/asan/bin/kudu with pid 17337
2026-06-20T14:52:07Z chronyd exiting
[ OK ] SecurityITest.TestEnableAndDisableEncryptedTSK (64910 ms)
[----------] 30 tests from SecurityITest (298080 ms total)
[----------] 6 tests from EncryptionPolicyTest
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/0
Loading random data
Initializing database '/tmp/dist-test-taskUci3DU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781966829091300-388-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 20 14:52:07 dist-test-slave-q9j6 krb5kdc[17522](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 20 14:52:07 dist-test-slave-q9j6 krb5kdc[17522](info): set up 2 sockets
Jun 20 14:52:07 dist-test-slave-q9j6 krb5kdc[17522](info): commencing operation
krb5kdc: starting...
W20260620 14:52:08.327876 388 mini_kdc.cc:121] Time spent starting KDC: real 1.073s user 0.007s sys 0.021s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskUci3DU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781966829091300-388-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskUci3DU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781966829091300-388-0/krb5kdc/test-user.keytab.
Jun 20 14:52:08 dist-test-slave-q9j6 krb5kdc[17522](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781967128, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-20T14:52:08Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T14:52:08Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.97.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskUci3DU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781966829091300-388-0/krb5kdc/kudu_127.0.97.62.keytab.
Entry for principal kudu/127.0.97.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskUci3DU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781966829091300-388-0/krb5kdc/kudu_127.0.97.62.keytab.
I20260620 14:52:08.582540 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskUci3DU/build/asan/bin/kudu
/tmp/dist-test-taskUci3DU/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskUci3DU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781966829091300-388-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskUci3DU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781966829091300-388-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskUci3DU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781966829091300-388-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskUci3DU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781966829091300-388-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:40169
--webserver_interface=127.0.97.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskUci3DU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781966829091300-388-0/krb5kdc/kudu_127.0.97.62.keytab
--principal=kudu/127.0.97.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:41589
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.97.62:40169
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskUci3DU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781966829091300-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskUci3DU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781966829091300-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskUci3DU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781966829091300-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 14:52:08.937314 17535 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 14:52:08.937616 17535 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 14:52:08.937785 17535 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 14:52:08.945124 17535 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 14:52:08.945235 17535 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 14:52:08.945329 17535 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 14:52:08.945432 17535 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 14:52:08.945508 17535 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 14:52:08.955672 17535 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:41589
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskUci3DU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781966829091300-388-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskUci3DU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781966829091300-388-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.97.62:40169
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskUci3DU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781966829091300-388-0/krb5kdc/kudu_127.0.97.62.keytab
--principal=kudu/127.0.97.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:40169
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskUci3DU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781966829091300-388-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.97.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskUci3DU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781966829091300-388-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 13:41:20 UTC on ccycloud.quasar-vdagoi.root.comops.site
build id 4457462
ASAN enabled
I20260620 14:52:08.957083 17535 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 14:52:08.958443 17535 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 14:52:08.971328 17540 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 14:52:08.971292 17543 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 14:52:08.971273 17541 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 14:52:08.971704 17535 server_base.cc:1034] running on GCE node
I20260620 14:52:08.972877 17535 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 14:52:08.974452 17535 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 14:52:08.975708 17535 hybrid_clock.cc:648] HybridClock initialized: now 1781967128975624 us; error 71 us; skew 500 ppm
I20260620 14:52:08.976044 17535 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 14:52:08 dist-test-slave-q9j6 krb5kdc[17522](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781967128, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 14:52:08.979343 17535 init.cc:377] Logged in from keytab as kudu/127.0.97.62@KRBTEST.COM (short username kudu)
I20260620 14:52:08.981096 17535 webserver.cc:460] Webserver started at http://127.0.97.62:39005/ using document root <none> and password file <none>
I20260620 14:52:08.981864 17535 fs_manager.cc:359] Metadata directory not provided
I20260620 14:52:08.982004 17535 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 14:52:08.982358 17535 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 14:52:08.985596 17535 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskUci3DU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781966829091300-388-0/minicluster-data/master-0/data/instance:
uuid: "aafe3ca9ef6d450592d1ad1386e36fa4"
format_stamp: "Formatted at 2026-06-20 14:52:08 on dist-test-slave-q9j6"
server_key: "7ea56e1cc5a3d7dbbca7cd597908b116"
server_key_iv: "eb710dcae481ec75e30e3c777622b22f"
server_key_version: "encryptionkey@0"
I20260620 14:52:08.986454 17535 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskUci3DU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781966829091300-388-0/minicluster-data/master-0/wal/instance:
uuid: "aafe3ca9ef6d450592d1ad1386e36fa4"
format_stamp: "Formatted at 2026-06-20 14:52:08 on dist-test-slave-q9j6"
server_key: "7ea56e1cc5a3d7dbbca7cd597908b116"
server_key_iv: "eb710dcae481ec75e30e3c777622b22f"
server_key_version: "encryptionkey@0"
I20260620 14:52:08.992182 17535 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.007s sys 0.001s
I20260620 14:52:08.995970 17550 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 14:52:08.997400 17535 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260620 14:52:08.997596 17535 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskUci3DU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781966829091300-388-0/minicluster-data/master-0/data,/tmp/dist-test-taskUci3DU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781966829091300-388-0/minicluster-data/master-0/wal
uuid: "aafe3ca9ef6d450592d1ad1386e36fa4"
format_stamp: "Formatted at 2026-06-20 14:52:08 on dist-test-slave-q9j6"
server_key: "7ea56e1cc5a3d7dbbca7cd597908b116"
server_key_iv: "eb710dcae481ec75e30e3c777622b22f"
server_key_version: "encryptionkey@0"
I20260620 14:52:08.997812 17535 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskUci3DU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781966829091300-388-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskUci3DU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781966829091300-388-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskUci3DU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781966829091300-388-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 14:52:09.019044 17535 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 14:52:09.023672 17535 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 14:52:09.023995 17535 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 14:52:09.044947 17535 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.62:40169
I20260620 14:52:09.044973 17601 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.62:40169 every 8 connection(s)
I20260620 14:52:09.046518 17535 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskUci3DU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781966829091300-388-0/minicluster-data/master-0/data/info.pb
I20260620 14:52:09.049166 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskUci3DU/build/asan/bin/kudu as pid 17535
I20260620 14:52:09.049402 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskUci3DU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781966829091300-388-0/minicluster-data/master-0/wal/instance
I20260620 14:52:09.050097 388 external_mini_cluster.cc:1428] Setting key 548f4436ef89fdf1968de77353229b3c
I20260620 14:52:09.050684 17602 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 14:52:09.066177 17602 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P aafe3ca9ef6d450592d1ad1386e36fa4: Bootstrap starting.
Jun 20 14:52:09 dist-test-slave-q9j6 krb5kdc[17522](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781967128, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260620 14:52:09.070268 17602 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P aafe3ca9ef6d450592d1ad1386e36fa4: Neither blocks nor log segments found. Creating new log.
I20260620 14:52:09.071743 17602 log.cc:824] T 00000000000000000000000000000000 P aafe3ca9ef6d450592d1ad1386e36fa4: Log is configured to *not* fsync() on all Append() calls
I20260620 14:52:09.075546 17602 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P aafe3ca9ef6d450592d1ad1386e36fa4: No bootstrap required, opened a new log
I20260620 14:52:09.077219 17605 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:52:09.052320 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42116 (local address 127.0.97.62:40169)
0620 14:52:09.053233 (+ 913us) server_negotiation.cc:206] Beginning negotiation
0620 14:52:09.053248 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0620 14:52:09.053312 (+ 64us) server_negotiation.cc:407] Connection header received
0620 14:52:09.055870 (+ 2558us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:52:09.055904 (+ 34us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:52:09.056536 (+ 632us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:52:09.057058 (+ 522us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:52:09.058492 (+ 1434us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:52:09.060544 (+ 2052us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:52:09.062036 (+ 1492us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:52:09.063228 (+ 1192us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:52:09.063457 (+ 229us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:52:09.067882 (+ 4425us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 14:52:09.067895 (+ 13us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 14:52:09.067920 (+ 25us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 14:52:09.067980 (+ 60us) server_negotiation.cc:922] calling sasl_server_start()
0620 14:52:09.072087 (+ 4107us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:52:09.073120 (+ 1033us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:52:09.073128 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:52:09.073135 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:52:09.073419 (+ 284us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:52:09.073841 (+ 422us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:52:09.073846 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:52:09.073849 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:52:09.074592 (+ 743us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 14:52:09.074894 (+ 302us) server_negotiation.cc:1035] Waiting for connection context
0620 14:52:09.075285 (+ 391us) server_negotiation.cc:299] Negotiation successful
0620 14:52:09.075786 (+ 501us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":642,"thread_start_us":386,"threads_started":1}
I20260620 14:52:09.084697 17602 raft_consensus.cc:348] T 00000000000000000000000000000000 P aafe3ca9ef6d450592d1ad1386e36fa4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "aafe3ca9ef6d450592d1ad1386e36fa4" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 40169 } }
I20260620 14:52:09.085104 17602 raft_consensus.cc:374] T 00000000000000000000000000000000 P aafe3ca9ef6d450592d1ad1386e36fa4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 14:52:09.085176 17602 raft_consensus.cc:729] T 00000000000000000000000000000000 P aafe3ca9ef6d450592d1ad1386e36fa4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: aafe3ca9ef6d450592d1ad1386e36fa4, State: Initialized, Role: FOLLOWER
I20260620 14:52:09.085753 17602 consensus_queue.cc:260] T 00000000000000000000000000000000 P aafe3ca9ef6d450592d1ad1386e36fa4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "aafe3ca9ef6d450592d1ad1386e36fa4" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 40169 } }
I20260620 14:52:09.085973 17602 raft_consensus.cc:388] T 00000000000000000000000000000000 P aafe3ca9ef6d450592d1ad1386e36fa4 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 14:52:09.086076 17602 raft_consensus.cc:482] T 00000000000000000000000000000000 P aafe3ca9ef6d450592d1ad1386e36fa4 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 14:52:09.086194 17602 raft_consensus.cc:3037] T 00000000000000000000000000000000 P aafe3ca9ef6d450592d1ad1386e36fa4 [term 0 FOLLOWER]: Advancing to term 1
I20260620 14:52:09.088471 17602 raft_consensus.cc:504] T 00000000000000000000000000000000 P aafe3ca9ef6d450592d1ad1386e36fa4 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "aafe3ca9ef6d450592d1ad1386e36fa4" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 40169 } }
I20260620 14:52:09.088964 17602 leader_election.cc:302] T 00000000000000000000000000000000 P aafe3ca9ef6d450592d1ad1386e36fa4 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: aafe3ca9ef6d450592d1ad1386e36fa4; no voters:
I20260620 14:52:09.089663 17602 leader_election.cc:288] T 00000000000000000000000000000000 P aafe3ca9ef6d450592d1ad1386e36fa4 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260620 14:52:09.089792 17607 raft_consensus.cc:2781] T 00000000000000000000000000000000 P aafe3ca9ef6d450592d1ad1386e36fa4 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 14:52:09.090740 17607 raft_consensus.cc:686] T 00000000000000000000000000000000 P aafe3ca9ef6d450592d1ad1386e36fa4 [term 1 LEADER]: Becoming Leader. State: Replica: aafe3ca9ef6d450592d1ad1386e36fa4, State: Running, Role: LEADER
I20260620 14:52:09.091367 17607 consensus_queue.cc:237] T 00000000000000000000000000000000 P aafe3ca9ef6d450592d1ad1386e36fa4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "aafe3ca9ef6d450592d1ad1386e36fa4" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 40169 } }
I20260620 14:52:09.092007 17602 sys_catalog.cc:564] T 00000000000000000000000000000000 P aafe3ca9ef6d450592d1ad1386e36fa4 [sys.catalog]: configured and running, proceeding with master startup.
I20260620 14:52:09.094679 17608 sys_catalog.cc:455] T 00000000000000000000000000000000 P aafe3ca9ef6d450592d1ad1386e36fa4 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "aafe3ca9ef6d450592d1ad1386e36fa4" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "aafe3ca9ef6d450592d1ad1386e36fa4" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 40169 } } }
I20260620 14:52:09.095106 17608 sys_catalog.cc:458] T 00000000000000000000000000000000 P aafe3ca9ef6d450592d1ad1386e36fa4 [sys.catalog]: This master's current role is: LEADER
I20260620 14:52:09.094527 17609 sys_catalog.cc:455] T 00000000000000000000000000000000 P aafe3ca9ef6d450592d1ad1386e36fa4 [sys.catalog]: SysCatalogTable state changed. Reason: New leader aafe3ca9ef6d450592d1ad1386e36fa4. Latest consensus state: current_term: 1 leader_uuid: "aafe3ca9ef6d450592d1ad1386e36fa4" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "aafe3ca9ef6d450592d1ad1386e36fa4" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 40169 } } }
I20260620 14:52:09.095799 17609 sys_catalog.cc:458] T 00000000000000000000000000000000 P aafe3ca9ef6d450592d1ad1386e36fa4 [sys.catalog]: This master's current role is: LEADER
W20260620 14:52:09.106081 17623 catalog_manager.cc:1558] T 00000000000000000000000000000000 P aafe3ca9ef6d450592d1ad1386e36fa4: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260620 14:52:09.106280 17623 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260620 14:52:09.106499 17616 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 14:52:09.108444 17616 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 14:52:09.117806 17616 catalog_manager.cc:1347] Generated new cluster ID: 9095b8fd83304ed19fa6e99566a33265
I20260620 14:52:09.117954 17616 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 14:52:09.135908 17616 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 14:52:09.137279 17616 catalog_manager.cc:1504] Loading token signing keys...
I20260620 14:52:09.147347 17616 catalog_manager.cc:5868] T 00000000000000000000000000000000 P aafe3ca9ef6d450592d1ad1386e36fa4: Generated new TSK 0
I20260620 14:52:09.148545 17616 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.0.97.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskUci3DU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781966829091300-388-0/krb5kdc/kudu_127.0.97.1.keytab.
Entry for principal kudu/127.0.97.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskUci3DU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781966829091300-388-0/krb5kdc/kudu_127.0.97.1.keytab.
I20260620 14:52:09.216562 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskUci3DU/build/asan/bin/kudu
/tmp/dist-test-taskUci3DU/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskUci3DU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781966829091300-388-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskUci3DU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781966829091300-388-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskUci3DU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781966829091300-388-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskUci3DU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781966829091300-388-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.1:0
--local_ip_for_outbound_sockets=127.0.97.1
--webserver_interface=127.0.97.1
--webserver_port=0
--tserver_master_addrs=127.0.97.62:40169
--keytab_file=/tmp/dist-test-taskUci3DU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781966829091300-388-0/krb5kdc/kudu_127.0.97.1.keytab
--principal=kudu/127.0.97.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:41589
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskUci3DU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781966829091300-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskUci3DU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781966829091300-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskUci3DU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781966829091300-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 14:52:09.562214 17628 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 14:52:09.562511 17628 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 14:52:09.562688 17628 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 14:52:09.570413 17628 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 14:52:09.570537 17628 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 14:52:09.570717 17628 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.1
I20260620 14:52:09.581305 17628 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:41589
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskUci3DU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781966829091300-388-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskUci3DU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781966829091300-388-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskUci3DU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781966829091300-388-0/krb5kdc/kudu_127.0.97.1.keytab
--principal=kudu/127.0.97.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskUci3DU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781966829091300-388-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.97.1
--webserver_port=0
--tserver_master_addrs=127.0.97.62:40169
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.1
--log_dir=/tmp/dist-test-taskUci3DU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781966829091300-388-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 13:41:20 UTC on ccycloud.quasar-vdagoi.root.comops.site
build id 4457462
ASAN enabled
I20260620 14:52:09.582726 17628 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 14:52:09.584179 17628 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 14:52:09.598073 17634 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 14:52:09.598034 17636 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 14:52:09.598615 17633 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 14:52:09.598973 17628 server_base.cc:1034] running on GCE node
I20260620 14:52:09.599694 17628 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 14:52:09.600781 17628 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 14:52:09.602023 17628 hybrid_clock.cc:648] HybridClock initialized: now 1781967129601977 us; error 66 us; skew 500 ppm
I20260620 14:52:09.602385 17628 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 14:52:09 dist-test-slave-q9j6 krb5kdc[17522](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781967129, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 14:52:09.605682 17628 init.cc:377] Logged in from keytab as kudu/127.0.97.1@KRBTEST.COM (short username kudu)
I20260620 14:52:09.607229 17628 webserver.cc:460] Webserver started at http://127.0.97.1:46119/ using document root <none> and password file <none>
I20260620 14:52:09.608089 17628 fs_manager.cc:359] Metadata directory not provided
I20260620 14:52:09.608244 17628 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 14:52:09.608577 17628 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 14:52:09.612452 17628 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskUci3DU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781966829091300-388-0/minicluster-data/ts-0/data/instance:
uuid: "b79dd7c833e24143a22520483886771d"
format_stamp: "Formatted at 2026-06-20 14:52:09 on dist-test-slave-q9j6"
server_key: "a064aab0262ccc8675b8fa1631e62938"
server_key_iv: "c14e418f2de9c50aff39d57d50b9a5c1"
server_key_version: "encryptionkey@0"
I20260620 14:52:09.613385 17628 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskUci3DU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781966829091300-388-0/minicluster-data/ts-0/wal/instance:
uuid: "b79dd7c833e24143a22520483886771d"
format_stamp: "Formatted at 2026-06-20 14:52:09 on dist-test-slave-q9j6"
server_key: "a064aab0262ccc8675b8fa1631e62938"
server_key_iv: "c14e418f2de9c50aff39d57d50b9a5c1"
server_key_version: "encryptionkey@0"
I20260620 14:52:09.618887 17628 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.000s
I20260620 14:52:09.622862 17643 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 14:52:09.624388 17628 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260620 14:52:09.624584 17628 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskUci3DU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781966829091300-388-0/minicluster-data/ts-0/data,/tmp/dist-test-taskUci3DU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781966829091300-388-0/minicluster-data/ts-0/wal
uuid: "b79dd7c833e24143a22520483886771d"
format_stamp: "Formatted at 2026-06-20 14:52:09 on dist-test-slave-q9j6"
server_key: "a064aab0262ccc8675b8fa1631e62938"
server_key_iv: "c14e418f2de9c50aff39d57d50b9a5c1"
server_key_version: "encryptionkey@0"
I20260620 14:52:09.624820 17628 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskUci3DU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781966829091300-388-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskUci3DU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781966829091300-388-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskUci3DU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781966829091300-388-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 14:52:09.643424 17628 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 14:52:09.648070 17628 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 14:52:09.648407 17628 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 14:52:09.649443 17628 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 14:52:09.650856 17628 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 14:52:09.650965 17628 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 14:52:09.651126 17628 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 14:52:09.651201 17628 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 14:52:09.684741 17628 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.1:36027
I20260620 14:52:09.684777 17755 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.1:36027 every 8 connection(s)
I20260620 14:52:09.686059 17628 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskUci3DU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781966829091300-388-0/minicluster-data/ts-0/data/info.pb
I20260620 14:52:09.692059 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskUci3DU/build/asan/bin/kudu as pid 17628
I20260620 14:52:09.692301 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskUci3DU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781966829091300-388-0/minicluster-data/ts-0/wal/instance
I20260620 14:52:09.692926 388 external_mini_cluster.cc:1428] Setting key 8a4e809a0c06e6ac5f92d03c1bcc0312
Jun 20 14:52:09 dist-test-slave-q9j6 krb5kdc[17522](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781967129, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.1@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260620 14:52:09.708848 17758 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:52:09.688727 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.1:53703 (local address 127.0.97.62:40169)
0620 14:52:09.689195 (+ 468us) server_negotiation.cc:206] Beginning negotiation
0620 14:52:09.689200 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 14:52:09.690071 (+ 871us) server_negotiation.cc:407] Connection header received
0620 14:52:09.692029 (+ 1958us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:52:09.692035 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:52:09.692132 (+ 97us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:52:09.692291 (+ 159us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:52:09.694156 (+ 1865us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:52:09.695048 (+ 892us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:52:09.696874 (+ 1826us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:52:09.697319 (+ 445us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:52:09.697473 (+ 154us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:52:09.701522 (+ 4049us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 14:52:09.701528 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 14:52:09.701531 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 14:52:09.701567 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0620 14:52:09.705254 (+ 3687us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:52:09.706095 (+ 841us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:52:09.706104 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:52:09.706110 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:52:09.706274 (+ 164us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:52:09.706705 (+ 431us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:52:09.706712 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:52:09.706718 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:52:09.707131 (+ 413us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 14:52:09.707289 (+ 158us) server_negotiation.cc:1035] Waiting for connection context
0620 14:52:09.708081 (+ 792us) server_negotiation.cc:299] Negotiation successful
0620 14:52:09.708345 (+ 264us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":3,"server-negotiator.queue_time_us":357,"thread_start_us":255,"threads_started":1}
I20260620 14:52:09.709466 17759 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:52:09.689159 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:40169 (local address 127.0.97.1:53703)
0620 14:52:09.689890 (+ 731us) negotiation.cc:107] Waiting for socket to connect
0620 14:52:09.689942 (+ 52us) client_negotiation.cc:174] Beginning negotiation
0620 14:52:09.691740 (+ 1798us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 14:52:09.692582 (+ 842us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 14:52:09.692591 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 14:52:09.693180 (+ 589us) client_negotiation.cc:189] Negotiated authn=SASL
0620 14:52:09.693972 (+ 792us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:52:09.693994 (+ 22us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:52:09.695232 (+ 1238us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:52:09.695236 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:52:09.696718 (+ 1482us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:52:09.696730 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:52:09.697576 (+ 846us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:52:09.697584 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:52:09.697846 (+ 262us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:52:09.698590 (+ 744us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 14:52:09.698618 (+ 28us) client_negotiation.cc:656] calling sasl_client_start()
0620 14:52:09.701270 (+ 2652us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 14:52:09.705436 (+ 4166us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 14:52:09.705443 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 14:52:09.705459 (+ 16us) client_negotiation.cc:760] calling sasl_client_step()
0620 14:52:09.705941 (+ 482us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 14:52:09.706386 (+ 445us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 14:52:09.706389 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 14:52:09.706391 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 14:52:09.706578 (+ 187us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 14:52:09.707254 (+ 676us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 14:52:09.707261 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 14:52:09.707659 (+ 398us) client_negotiation.cc:769] Sending connection context
0620 14:52:09.707922 (+ 263us) client_negotiation.cc:240] Negotiation successful
0620 14:52:09.708324 (+ 402us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":500,"thread_start_us":291,"threads_started":1}
I20260620 14:52:09.711089 17756 heartbeater.cc:347] Connected to a master server at 127.0.97.62:40169
I20260620 14:52:09.711426 17756 heartbeater.cc:464] Registering TS with master...
I20260620 14:52:09.712390 17756 heartbeater.cc:511] Master 127.0.97.62:40169 requested a full tablet report, sending...
I20260620 14:52:09.714790 17567 ts_manager.cc:194] Registered new tserver with Master: b79dd7c833e24143a22520483886771d (127.0.97.1:36027)
I20260620 14:52:09.716437 17567 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.97.1@KRBTEST.COM'} at 127.0.97.1:53703
WARNING: no policy specified for kudu/127.0.97.2@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskUci3DU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781966829091300-388-0/krb5kdc/kudu_127.0.97.2.keytab.
Entry for principal kudu/127.0.97.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskUci3DU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781966829091300-388-0/krb5kdc/kudu_127.0.97.2.keytab.
I20260620 14:52:09.749003 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskUci3DU/build/asan/bin/kudu
/tmp/dist-test-taskUci3DU/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskUci3DU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781966829091300-388-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskUci3DU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781966829091300-388-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskUci3DU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781966829091300-388-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskUci3DU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781966829091300-388-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.2:0
--local_ip_for_outbound_sockets=127.0.97.2
--webserver_interface=127.0.97.2
--webserver_port=0
--tserver_master_addrs=127.0.97.62:40169
--keytab_file=/tmp/dist-test-taskUci3DU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781966829091300-388-0/krb5kdc/kudu_127.0.97.2.keytab
--principal=kudu/127.0.97.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:41589
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskUci3DU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781966829091300-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskUci3DU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781966829091300-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskUci3DU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781966829091300-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 14:52:10.090538 17762 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 14:52:10.090903 17762 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 14:52:10.091080 17762 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 14:52:10.098771 17762 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 14:52:10.098929 17762 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 14:52:10.099112 17762 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.2
I20260620 14:52:10.109344 17762 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:41589
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskUci3DU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781966829091300-388-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskUci3DU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781966829091300-388-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskUci3DU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781966829091300-388-0/krb5kdc/kudu_127.0.97.2.keytab
--principal=kudu/127.0.97.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskUci3DU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781966829091300-388-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.97.2
--webserver_port=0
--tserver_master_addrs=127.0.97.62:40169
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.2
--log_dir=/tmp/dist-test-taskUci3DU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781966829091300-388-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 13:41:20 UTC on ccycloud.quasar-vdagoi.root.comops.site
build id 4457462
ASAN enabled
I20260620 14:52:10.110733 17762 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 14:52:10.112231 17762 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 14:52:10.125984 17767 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 14:52:10.126035 17768 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 14:52:10.125926 17770 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 14:52:10.127687 17762 server_base.cc:1034] running on GCE node
I20260620 14:52:10.128392 17762 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 14:52:10.129379 17762 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 14:52:10.130614 17762 hybrid_clock.cc:648] HybridClock initialized: now 1781967130130562 us; error 50 us; skew 500 ppm
I20260620 14:52:10.130916 17762 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 14:52:10 dist-test-slave-q9j6 krb5kdc[17522](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781967130, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 14:52:10.134140 17762 init.cc:377] Logged in from keytab as kudu/127.0.97.2@KRBTEST.COM (short username kudu)
I20260620 14:52:10.135694 17762 webserver.cc:460] Webserver started at http://127.0.97.2:38727/ using document root <none> and password file <none>
I20260620 14:52:10.136415 17762 fs_manager.cc:359] Metadata directory not provided
I20260620 14:52:10.136519 17762 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 14:52:10.136785 17762 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 14:52:10.140005 17762 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskUci3DU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781966829091300-388-0/minicluster-data/ts-1/data/instance:
uuid: "43a6e0a26107471fa26598ca375dbdbc"
format_stamp: "Formatted at 2026-06-20 14:52:10 on dist-test-slave-q9j6"
server_key: "81d48b960dcd38932ac4f18093799f88"
server_key_iv: "3b6f25a6c0d560258983d649afd2ed91"
server_key_version: "encryptionkey@0"
I20260620 14:52:10.140817 17762 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskUci3DU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781966829091300-388-0/minicluster-data/ts-1/wal/instance:
uuid: "43a6e0a26107471fa26598ca375dbdbc"
format_stamp: "Formatted at 2026-06-20 14:52:10 on dist-test-slave-q9j6"
server_key: "81d48b960dcd38932ac4f18093799f88"
server_key_iv: "3b6f25a6c0d560258983d649afd2ed91"
server_key_version: "encryptionkey@0"
I20260620 14:52:10.146306 17762 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260620 14:52:10.149981 17777 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 14:52:10.151407 17762 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.000s sys 0.002s
I20260620 14:52:10.151667 17762 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskUci3DU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781966829091300-388-0/minicluster-data/ts-1/data,/tmp/dist-test-taskUci3DU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781966829091300-388-0/minicluster-data/ts-1/wal
uuid: "43a6e0a26107471fa26598ca375dbdbc"
format_stamp: "Formatted at 2026-06-20 14:52:10 on dist-test-slave-q9j6"
server_key: "81d48b960dcd38932ac4f18093799f88"
server_key_iv: "3b6f25a6c0d560258983d649afd2ed91"
server_key_version: "encryptionkey@0"
I20260620 14:52:10.151901 17762 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskUci3DU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781966829091300-388-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskUci3DU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781966829091300-388-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskUci3DU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781966829091300-388-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 14:52:10.170898 17762 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 14:52:10.175550 17762 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 14:52:10.175892 17762 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 14:52:10.176945 17762 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 14:52:10.178449 17762 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 14:52:10.178560 17762 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 14:52:10.178730 17762 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 14:52:10.178802 17762 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 14:52:10.213034 17762 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.2:41611
I20260620 14:52:10.213071 17889 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.2:41611 every 8 connection(s)
I20260620 14:52:10.214380 17762 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskUci3DU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781966829091300-388-0/minicluster-data/ts-1/data/info.pb
I20260620 14:52:10.224151 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskUci3DU/build/asan/bin/kudu as pid 17762
I20260620 14:52:10.224363 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskUci3DU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781966829091300-388-0/minicluster-data/ts-1/wal/instance
I20260620 14:52:10.224818 388 external_mini_cluster.cc:1428] Setting key abfea1bc27e712b900eedbaab953b5a2
Jun 20 14:52:10 dist-test-slave-q9j6 krb5kdc[17522](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781967130, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.2@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260620 14:52:10.236960 17892 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:52:10.217112 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.2:50661 (local address 127.0.97.62:40169)
0620 14:52:10.217568 (+ 456us) server_negotiation.cc:206] Beginning negotiation
0620 14:52:10.217573 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 14:52:10.218494 (+ 921us) server_negotiation.cc:407] Connection header received
0620 14:52:10.220535 (+ 2041us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:52:10.220541 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:52:10.220639 (+ 98us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:52:10.220766 (+ 127us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:52:10.222468 (+ 1702us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:52:10.223185 (+ 717us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:52:10.225201 (+ 2016us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:52:10.225804 (+ 603us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:52:10.225966 (+ 162us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:52:10.230384 (+ 4418us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 14:52:10.230389 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 14:52:10.230392 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 14:52:10.230429 (+ 37us) server_negotiation.cc:922] calling sasl_server_start()
0620 14:52:10.233228 (+ 2799us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:52:10.234071 (+ 843us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:52:10.234082 (+ 11us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:52:10.234085 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:52:10.234237 (+ 152us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:52:10.234732 (+ 495us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:52:10.234739 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:52:10.234743 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:52:10.235184 (+ 441us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 14:52:10.235363 (+ 179us) server_negotiation.cc:1035] Waiting for connection context
0620 14:52:10.236171 (+ 808us) server_negotiation.cc:299] Negotiation successful
0620 14:52:10.236459 (+ 288us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":346,"thread_start_us":242,"threads_started":1}
I20260620 14:52:10.237643 17893 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:52:10.217617 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:40169 (local address 127.0.97.2:50661)
0620 14:52:10.218323 (+ 706us) negotiation.cc:107] Waiting for socket to connect
0620 14:52:10.218366 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0620 14:52:10.220209 (+ 1843us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 14:52:10.221045 (+ 836us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 14:52:10.221056 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 14:52:10.221625 (+ 569us) client_negotiation.cc:189] Negotiated authn=SASL
0620 14:52:10.222263 (+ 638us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:52:10.222283 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:52:10.223369 (+ 1086us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:52:10.223374 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:52:10.225035 (+ 1661us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:52:10.225050 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:52:10.226080 (+ 1030us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:52:10.226085 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:52:10.226305 (+ 220us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:52:10.227139 (+ 834us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 14:52:10.227181 (+ 42us) client_negotiation.cc:656] calling sasl_client_start()
0620 14:52:10.230114 (+ 2933us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 14:52:10.233410 (+ 3296us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 14:52:10.233420 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 14:52:10.233441 (+ 21us) client_negotiation.cc:760] calling sasl_client_step()
0620 14:52:10.233920 (+ 479us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 14:52:10.234390 (+ 470us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 14:52:10.234394 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 14:52:10.234397 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0620 14:52:10.234598 (+ 201us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 14:52:10.235321 (+ 723us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 14:52:10.235328 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 14:52:10.235756 (+ 428us) client_negotiation.cc:769] Sending connection context
0620 14:52:10.236017 (+ 261us) client_negotiation.cc:240] Negotiation successful
0620 14:52:10.236457 (+ 440us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":471,"thread_start_us":277,"threads_started":1}
I20260620 14:52:10.239274 17890 heartbeater.cc:347] Connected to a master server at 127.0.97.62:40169
I20260620 14:52:10.239668 17890 heartbeater.cc:464] Registering TS with master...
I20260620 14:52:10.240464 17890 heartbeater.cc:511] Master 127.0.97.62:40169 requested a full tablet report, sending...
I20260620 14:52:10.242100 17567 ts_manager.cc:194] Registered new tserver with Master: 43a6e0a26107471fa26598ca375dbdbc (127.0.97.2:41611)
I20260620 14:52:10.242970 17567 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.97.2@KRBTEST.COM'} at 127.0.97.2:50661
WARNING: no policy specified for kudu/127.0.97.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskUci3DU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781966829091300-388-0/krb5kdc/kudu_127.0.97.3.keytab.
Entry for principal kudu/127.0.97.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskUci3DU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781966829091300-388-0/krb5kdc/kudu_127.0.97.3.keytab.
I20260620 14:52:10.281239 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskUci3DU/build/asan/bin/kudu
/tmp/dist-test-taskUci3DU/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskUci3DU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781966829091300-388-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskUci3DU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781966829091300-388-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskUci3DU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781966829091300-388-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskUci3DU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781966829091300-388-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.3:0
--local_ip_for_outbound_sockets=127.0.97.3
--webserver_interface=127.0.97.3
--webserver_port=0
--tserver_master_addrs=127.0.97.62:40169
--keytab_file=/tmp/dist-test-taskUci3DU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781966829091300-388-0/krb5kdc/kudu_127.0.97.3.keytab
--principal=kudu/127.0.97.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:41589
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskUci3DU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781966829091300-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskUci3DU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781966829091300-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskUci3DU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781966829091300-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 14:52:10.629891 17896 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 14:52:10.630213 17896 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 14:52:10.630414 17896 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 14:52:10.638309 17896 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 14:52:10.638444 17896 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 14:52:10.638626 17896 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.3
I20260620 14:52:10.649205 17896 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:41589
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskUci3DU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781966829091300-388-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskUci3DU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781966829091300-388-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskUci3DU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781966829091300-388-0/krb5kdc/kudu_127.0.97.3.keytab
--principal=kudu/127.0.97.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskUci3DU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781966829091300-388-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.97.3
--webserver_port=0
--tserver_master_addrs=127.0.97.62:40169
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.3
--log_dir=/tmp/dist-test-taskUci3DU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781966829091300-388-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 13:41:20 UTC on ccycloud.quasar-vdagoi.root.comops.site
build id 4457462
ASAN enabled
I20260620 14:52:10.650669 17896 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 14:52:10.652161 17896 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 14:52:10.666515 17901 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 14:52:10.666615 17904 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 14:52:10.666636 17902 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 14:52:10.667524 17896 server_base.cc:1034] running on GCE node
I20260620 14:52:10.668262 17896 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 14:52:10.669342 17896 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 14:52:10.670612 17896 hybrid_clock.cc:648] HybridClock initialized: now 1781967130670542 us; error 65 us; skew 500 ppm
I20260620 14:52:10.670980 17896 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 14:52:10 dist-test-slave-q9j6 krb5kdc[17522](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781967130, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 14:52:10.674265 17896 init.cc:377] Logged in from keytab as kudu/127.0.97.3@KRBTEST.COM (short username kudu)
I20260620 14:52:10.675941 17896 webserver.cc:460] Webserver started at http://127.0.97.3:38113/ using document root <none> and password file <none>
I20260620 14:52:10.676725 17896 fs_manager.cc:359] Metadata directory not provided
I20260620 14:52:10.676894 17896 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 14:52:10.677240 17896 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 14:52:10.680620 17896 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskUci3DU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781966829091300-388-0/minicluster-data/ts-2/data/instance:
uuid: "85e2888b517740eda27f2796c008948b"
format_stamp: "Formatted at 2026-06-20 14:52:10 on dist-test-slave-q9j6"
server_key: "8a6365009243e39d67d2c69b84538c25"
server_key_iv: "cb814bcadb62b2a7a0b280154c81eb4b"
server_key_version: "encryptionkey@0"
I20260620 14:52:10.681532 17896 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskUci3DU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781966829091300-388-0/minicluster-data/ts-2/wal/instance:
uuid: "85e2888b517740eda27f2796c008948b"
format_stamp: "Formatted at 2026-06-20 14:52:10 on dist-test-slave-q9j6"
server_key: "8a6365009243e39d67d2c69b84538c25"
server_key_iv: "cb814bcadb62b2a7a0b280154c81eb4b"
server_key_version: "encryptionkey@0"
I20260620 14:52:10.688508 17896 fs_manager.cc:691] Time spent creating directory manager: real 0.007s user 0.004s sys 0.003s
I20260620 14:52:10.693300 17911 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 14:52:10.694916 17896 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.005s sys 0.000s
I20260620 14:52:10.695137 17896 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskUci3DU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781966829091300-388-0/minicluster-data/ts-2/data,/tmp/dist-test-taskUci3DU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781966829091300-388-0/minicluster-data/ts-2/wal
uuid: "85e2888b517740eda27f2796c008948b"
format_stamp: "Formatted at 2026-06-20 14:52:10 on dist-test-slave-q9j6"
server_key: "8a6365009243e39d67d2c69b84538c25"
server_key_iv: "cb814bcadb62b2a7a0b280154c81eb4b"
server_key_version: "encryptionkey@0"
I20260620 14:52:10.695379 17896 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskUci3DU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781966829091300-388-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskUci3DU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781966829091300-388-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskUci3DU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781966829091300-388-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 14:52:10.711103 17896 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 14:52:10.715721 17896 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 14:52:10.716049 17896 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 14:52:10.717121 17896 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 14:52:10.718640 17896 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 14:52:10.718762 17896 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 14:52:10.718922 17896 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 14:52:10.719002 17896 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 14:52:10.720520 17756 heartbeater.cc:503] Master 127.0.97.62:40169 was elected leader, sending a full tablet report...
I20260620 14:52:10.756614 17896 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.3:32787
I20260620 14:52:10.756650 18023 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.3:32787 every 8 connection(s)
I20260620 14:52:10.758066 17896 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskUci3DU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781966829091300-388-0/minicluster-data/ts-2/data/info.pb
I20260620 14:52:10.766800 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskUci3DU/build/asan/bin/kudu as pid 17896
I20260620 14:52:10.766995 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskUci3DU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781966829091300-388-0/minicluster-data/ts-2/wal/instance
I20260620 14:52:10.767460 388 external_mini_cluster.cc:1428] Setting key a0494f2ab869c9b74df8ecb1ae79a60f
Jun 20 14:52:10 dist-test-slave-q9j6 krb5kdc[17522](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781967130, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.3@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260620 14:52:10.780172 18026 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:52:10.760815 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.3:45579 (local address 127.0.97.62:40169)
0620 14:52:10.761265 (+ 450us) server_negotiation.cc:206] Beginning negotiation
0620 14:52:10.761269 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 14:52:10.762381 (+ 1112us) server_negotiation.cc:407] Connection header received
0620 14:52:10.764388 (+ 2007us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:52:10.764394 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:52:10.764504 (+ 110us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:52:10.764632 (+ 128us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:52:10.766248 (+ 1616us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:52:10.767155 (+ 907us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:52:10.769280 (+ 2125us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:52:10.769813 (+ 533us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:52:10.769951 (+ 138us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:52:10.773820 (+ 3869us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 14:52:10.773826 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 14:52:10.773830 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 14:52:10.773868 (+ 38us) server_negotiation.cc:922] calling sasl_server_start()
0620 14:52:10.776513 (+ 2645us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:52:10.777416 (+ 903us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:52:10.777422 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:52:10.777425 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:52:10.777557 (+ 132us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:52:10.778053 (+ 496us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:52:10.778057 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:52:10.778059 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:52:10.778361 (+ 302us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 14:52:10.778531 (+ 170us) server_negotiation.cc:1035] Waiting for connection context
0620 14:52:10.779455 (+ 924us) server_negotiation.cc:299] Negotiation successful
0620 14:52:10.779774 (+ 319us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":337,"thread_start_us":207,"threads_started":1}
I20260620 14:52:10.781198 18027 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:52:10.761420 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:40169 (local address 127.0.97.3:45579)
0620 14:52:10.762199 (+ 779us) negotiation.cc:107] Waiting for socket to connect
0620 14:52:10.762245 (+ 46us) client_negotiation.cc:174] Beginning negotiation
0620 14:52:10.764082 (+ 1837us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 14:52:10.764896 (+ 814us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 14:52:10.764905 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 14:52:10.765469 (+ 564us) client_negotiation.cc:189] Negotiated authn=SASL
0620 14:52:10.766061 (+ 592us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:52:10.766079 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:52:10.767312 (+ 1233us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:52:10.767318 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:52:10.769123 (+ 1805us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:52:10.769137 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:52:10.770079 (+ 942us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:52:10.770089 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:52:10.770344 (+ 255us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:52:10.771095 (+ 751us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 14:52:10.771122 (+ 27us) client_negotiation.cc:656] calling sasl_client_start()
0620 14:52:10.773536 (+ 2414us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 14:52:10.776714 (+ 3178us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 14:52:10.776725 (+ 11us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 14:52:10.776752 (+ 27us) client_negotiation.cc:760] calling sasl_client_step()
0620 14:52:10.777248 (+ 496us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 14:52:10.777687 (+ 439us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 14:52:10.777692 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 14:52:10.777696 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0620 14:52:10.777913 (+ 217us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 14:52:10.778505 (+ 592us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 14:52:10.778514 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 14:52:10.778992 (+ 478us) client_negotiation.cc:769] Sending connection context
0620 14:52:10.779294 (+ 302us) client_negotiation.cc:240] Negotiation successful
0620 14:52:10.779942 (+ 648us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":525,"thread_start_us":336,"threads_started":1}
I20260620 14:52:10.782810 18024 heartbeater.cc:347] Connected to a master server at 127.0.97.62:40169
I20260620 14:52:10.783238 18024 heartbeater.cc:464] Registering TS with master...
I20260620 14:52:10.784139 18024 heartbeater.cc:511] Master 127.0.97.62:40169 requested a full tablet report, sending...
I20260620 14:52:10.785727 17567 ts_manager.cc:194] Registered new tserver with Master: 85e2888b517740eda27f2796c008948b (127.0.97.3:32787)
I20260620 14:52:10.786595 17567 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.97.3@KRBTEST.COM'} at 127.0.97.3:45579
I20260620 14:52:10.796039 388 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260620 14:52:10.815158 18026 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:52:10.801298 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:32930 (local address 127.0.97.62:40169)
0620 14:52:10.801526 (+ 228us) server_negotiation.cc:206] Beginning negotiation
0620 14:52:10.801532 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 14:52:10.802044 (+ 512us) server_negotiation.cc:407] Connection header received
0620 14:52:10.802204 (+ 160us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:52:10.802208 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:52:10.802296 (+ 88us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:52:10.802394 (+ 98us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:52:10.803984 (+ 1590us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:52:10.804806 (+ 822us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:52:10.806225 (+ 1419us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:52:10.806658 (+ 433us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:52:10.806772 (+ 114us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:52:10.809620 (+ 2848us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 14:52:10.809626 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 14:52:10.809630 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 14:52:10.809704 (+ 74us) server_negotiation.cc:922] calling sasl_server_start()
0620 14:52:10.812566 (+ 2862us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:52:10.813308 (+ 742us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:52:10.813312 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:52:10.813314 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:52:10.813473 (+ 159us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:52:10.813824 (+ 351us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:52:10.813827 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:52:10.813829 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:52:10.814162 (+ 333us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 14:52:10.814385 (+ 223us) server_negotiation.cc:1035] Waiting for connection context
0620 14:52:10.814705 (+ 320us) server_negotiation.cc:299] Negotiation successful
0620 14:52:10.814912 (+ 207us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":65}
I20260620 14:52:10.819818 17567 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:32930:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260620 14:52:10.827028 17567 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260620 14:52:10.866767 18040 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:52:10.842016 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48046 (local address 127.0.97.1:36027)
0620 14:52:10.857168 (+ 15152us) server_negotiation.cc:206] Beginning negotiation
0620 14:52:10.857182 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0620 14:52:10.857223 (+ 41us) server_negotiation.cc:407] Connection header received
0620 14:52:10.857363 (+ 140us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:52:10.857375 (+ 12us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:52:10.857724 (+ 349us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:52:10.857995 (+ 271us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 14:52:10.858686 (+ 691us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:52:10.860698 (+ 2012us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:52:10.864021 (+ 3323us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:52:10.865606 (+ 1585us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:52:10.865740 (+ 134us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:52:10.865998 (+ 258us) server_negotiation.cc:1035] Waiting for connection context
0620 14:52:10.866226 (+ 228us) server_negotiation.cc:299] Negotiation successful
0620 14:52:10.866422 (+ 196us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":14903,"thread_start_us":16230,"threads_started":1}
I20260620 14:52:10.866851 18039 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:52:10.841324 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:36027 (local address 127.0.0.1:48046)
0620 14:52:10.856800 (+ 15476us) negotiation.cc:107] Waiting for socket to connect
0620 14:52:10.856828 (+ 28us) client_negotiation.cc:174] Beginning negotiation
0620 14:52:10.856940 (+ 112us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 14:52:10.858024 (+ 1084us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 14:52:10.858032 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 14:52:10.858064 (+ 32us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 14:52:10.858485 (+ 421us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:52:10.858505 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:52:10.860901 (+ 2396us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:52:10.860910 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:52:10.863803 (+ 2893us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:52:10.863820 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:52:10.865791 (+ 1971us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:52:10.865796 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:52:10.865937 (+ 141us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:52:10.865984 (+ 47us) client_negotiation.cc:769] Sending connection context
0620 14:52:10.866152 (+ 168us) client_negotiation.cc:240] Negotiation successful
0620 14:52:10.866432 (+ 280us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":15342,"thread_start_us":15155,"threads_started":1}
I20260620 14:52:10.870695 17691 tablet_service.cc:1467] Processing CreateTablet for tablet 1fccf291fe5c46708e3c84c1d2d05851 (DEFAULT_TABLE table=test-table [id=35790d800f0c4444864a5875107175c3]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 14:52:10.872417 17691 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1fccf291fe5c46708e3c84c1d2d05851. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 14:52:10.873840 18041 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:52:10.848840 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56740 (local address 127.0.97.2:41611)
0620 14:52:10.864107 (+ 15267us) server_negotiation.cc:206] Beginning negotiation
0620 14:52:10.864116 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0620 14:52:10.864147 (+ 31us) server_negotiation.cc:407] Connection header received
0620 14:52:10.864246 (+ 99us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:52:10.864254 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:52:10.864549 (+ 295us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:52:10.864810 (+ 261us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 14:52:10.866385 (+ 1575us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:52:10.867975 (+ 1590us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:52:10.871164 (+ 3189us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:52:10.872730 (+ 1566us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:52:10.872902 (+ 172us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:52:10.873099 (+ 197us) server_negotiation.cc:1035] Waiting for connection context
0620 14:52:10.873366 (+ 267us) server_negotiation.cc:299] Negotiation successful
0620 14:52:10.873531 (+ 165us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":15075,"thread_start_us":14955,"threads_started":1}
I20260620 14:52:10.873884 18042 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:52:10.854259 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.2:41611 (local address 127.0.0.1:56740)
0620 14:52:10.863246 (+ 8987us) negotiation.cc:107] Waiting for socket to connect
0620 14:52:10.863277 (+ 31us) client_negotiation.cc:174] Beginning negotiation
0620 14:52:10.863402 (+ 125us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 14:52:10.864784 (+ 1382us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 14:52:10.864789 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 14:52:10.864811 (+ 22us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 14:52:10.865267 (+ 456us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:52:10.865279 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:52:10.868117 (+ 2838us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:52:10.868122 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:52:10.870947 (+ 2825us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:52:10.870967 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:52:10.873031 (+ 2064us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:52:10.873036 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:52:10.873160 (+ 124us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:52:10.873187 (+ 27us) client_negotiation.cc:769] Sending connection context
0620 14:52:10.873302 (+ 115us) client_negotiation.cc:240] Negotiation successful
0620 14:52:10.873451 (+ 149us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":8823,"mutex_wait_us":191,"thread_start_us":9210,"threads_started":1}
I20260620 14:52:10.875850 18037 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:52:10.840980 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.3:32787 (local address 127.0.0.1:50330)
0620 14:52:10.847883 (+ 6903us) negotiation.cc:107] Waiting for socket to connect
0620 14:52:10.847951 (+ 68us) client_negotiation.cc:174] Beginning negotiation
0620 14:52:10.848166 (+ 215us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 14:52:10.866072 (+ 17906us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 14:52:10.866078 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 14:52:10.866108 (+ 30us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 14:52:10.866476 (+ 368us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:52:10.866487 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:52:10.868195 (+ 1708us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:52:10.868200 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:52:10.870947 (+ 2747us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:52:10.870967 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:52:10.875114 (+ 4147us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:52:10.875122 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:52:10.875298 (+ 176us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:52:10.875323 (+ 25us) client_negotiation.cc:769] Sending connection context
0620 14:52:10.875395 (+ 72us) client_negotiation.cc:240] Negotiation successful
0620 14:52:10.875502 (+ 107us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":6486,"mutex_wait_us":118,"thread_start_us":6297,"threads_started":1}
I20260620 14:52:10.878252 17825 tablet_service.cc:1467] Processing CreateTablet for tablet 1fccf291fe5c46708e3c84c1d2d05851 (DEFAULT_TABLE table=test-table [id=35790d800f0c4444864a5875107175c3]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 14:52:10.879992 18038 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:52:10.841092 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50330 (local address 127.0.97.3:32787)
0620 14:52:10.865451 (+ 24359us) server_negotiation.cc:206] Beginning negotiation
0620 14:52:10.865459 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0620 14:52:10.865494 (+ 35us) server_negotiation.cc:407] Connection header received
0620 14:52:10.865591 (+ 97us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:52:10.865599 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:52:10.865842 (+ 243us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:52:10.866058 (+ 216us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 14:52:10.866682 (+ 624us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:52:10.868017 (+ 1335us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:52:10.873202 (+ 5185us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:52:10.874888 (+ 1686us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:52:10.875027 (+ 139us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:52:10.879417 (+ 4390us) server_negotiation.cc:1035] Waiting for connection context
0620 14:52:10.879497 (+ 80us) server_negotiation.cc:299] Negotiation successful
0620 14:52:10.879721 (+ 224us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":24176,"thread_start_us":24324,"threads_started":1}
I20260620 14:52:10.879998 17825 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1fccf291fe5c46708e3c84c1d2d05851. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 14:52:10.883860 17959 tablet_service.cc:1467] Processing CreateTablet for tablet 1fccf291fe5c46708e3c84c1d2d05851 (DEFAULT_TABLE table=test-table [id=35790d800f0c4444864a5875107175c3]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 14:52:10.885219 17959 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1fccf291fe5c46708e3c84c1d2d05851. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 14:52:10.888335 18043 tablet_bootstrap.cc:492] T 1fccf291fe5c46708e3c84c1d2d05851 P b79dd7c833e24143a22520483886771d: Bootstrap starting.
I20260620 14:52:10.892477 18043 tablet_bootstrap.cc:654] T 1fccf291fe5c46708e3c84c1d2d05851 P b79dd7c833e24143a22520483886771d: Neither blocks nor log segments found. Creating new log.
I20260620 14:52:10.894052 18043 log.cc:824] T 1fccf291fe5c46708e3c84c1d2d05851 P b79dd7c833e24143a22520483886771d: Log is configured to *not* fsync() on all Append() calls
I20260620 14:52:10.894693 18044 tablet_bootstrap.cc:492] T 1fccf291fe5c46708e3c84c1d2d05851 P 43a6e0a26107471fa26598ca375dbdbc: Bootstrap starting.
I20260620 14:52:10.898593 18044 tablet_bootstrap.cc:654] T 1fccf291fe5c46708e3c84c1d2d05851 P 43a6e0a26107471fa26598ca375dbdbc: Neither blocks nor log segments found. Creating new log.
I20260620 14:52:10.898929 18043 tablet_bootstrap.cc:492] T 1fccf291fe5c46708e3c84c1d2d05851 P b79dd7c833e24143a22520483886771d: No bootstrap required, opened a new log
I20260620 14:52:10.899286 18043 ts_tablet_manager.cc:1397] T 1fccf291fe5c46708e3c84c1d2d05851 P b79dd7c833e24143a22520483886771d: Time spent bootstrapping tablet: real 0.011s user 0.003s sys 0.005s
I20260620 14:52:10.900187 18044 log.cc:824] T 1fccf291fe5c46708e3c84c1d2d05851 P 43a6e0a26107471fa26598ca375dbdbc: Log is configured to *not* fsync() on all Append() calls
I20260620 14:52:10.900760 18046 tablet_bootstrap.cc:492] T 1fccf291fe5c46708e3c84c1d2d05851 P 85e2888b517740eda27f2796c008948b: Bootstrap starting.
I20260620 14:52:10.906051 18046 tablet_bootstrap.cc:654] T 1fccf291fe5c46708e3c84c1d2d05851 P 85e2888b517740eda27f2796c008948b: Neither blocks nor log segments found. Creating new log.
I20260620 14:52:10.908004 18046 log.cc:824] T 1fccf291fe5c46708e3c84c1d2d05851 P 85e2888b517740eda27f2796c008948b: Log is configured to *not* fsync() on all Append() calls
I20260620 14:52:10.908591 18043 raft_consensus.cc:348] T 1fccf291fe5c46708e3c84c1d2d05851 P b79dd7c833e24143a22520483886771d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85e2888b517740eda27f2796c008948b" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 32787 } } peers { permanent_uuid: "b79dd7c833e24143a22520483886771d" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 36027 } } peers { permanent_uuid: "43a6e0a26107471fa26598ca375dbdbc" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 41611 } }
I20260620 14:52:10.909076 18043 raft_consensus.cc:374] T 1fccf291fe5c46708e3c84c1d2d05851 P b79dd7c833e24143a22520483886771d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 14:52:10.909173 18043 raft_consensus.cc:729] T 1fccf291fe5c46708e3c84c1d2d05851 P b79dd7c833e24143a22520483886771d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b79dd7c833e24143a22520483886771d, State: Initialized, Role: FOLLOWER
I20260620 14:52:10.909955 18043 consensus_queue.cc:260] T 1fccf291fe5c46708e3c84c1d2d05851 P b79dd7c833e24143a22520483886771d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85e2888b517740eda27f2796c008948b" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 32787 } } peers { permanent_uuid: "b79dd7c833e24143a22520483886771d" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 36027 } } peers { permanent_uuid: "43a6e0a26107471fa26598ca375dbdbc" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 41611 } }
I20260620 14:52:10.910955 18044 tablet_bootstrap.cc:492] T 1fccf291fe5c46708e3c84c1d2d05851 P 43a6e0a26107471fa26598ca375dbdbc: No bootstrap required, opened a new log
I20260620 14:52:10.911293 18044 ts_tablet_manager.cc:1397] T 1fccf291fe5c46708e3c84c1d2d05851 P 43a6e0a26107471fa26598ca375dbdbc: Time spent bootstrapping tablet: real 0.017s user 0.004s sys 0.009s
I20260620 14:52:10.912063 18043 ts_tablet_manager.cc:1428] T 1fccf291fe5c46708e3c84c1d2d05851 P b79dd7c833e24143a22520483886771d: Time spent starting tablet: real 0.013s user 0.009s sys 0.005s
I20260620 14:52:10.912249 18046 tablet_bootstrap.cc:492] T 1fccf291fe5c46708e3c84c1d2d05851 P 85e2888b517740eda27f2796c008948b: No bootstrap required, opened a new log
I20260620 14:52:10.912518 18046 ts_tablet_manager.cc:1397] T 1fccf291fe5c46708e3c84c1d2d05851 P 85e2888b517740eda27f2796c008948b: Time spent bootstrapping tablet: real 0.012s user 0.009s sys 0.002s
I20260620 14:52:10.922715 18044 raft_consensus.cc:348] T 1fccf291fe5c46708e3c84c1d2d05851 P 43a6e0a26107471fa26598ca375dbdbc [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85e2888b517740eda27f2796c008948b" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 32787 } } peers { permanent_uuid: "b79dd7c833e24143a22520483886771d" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 36027 } } peers { permanent_uuid: "43a6e0a26107471fa26598ca375dbdbc" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 41611 } }
I20260620 14:52:10.923297 18044 raft_consensus.cc:374] T 1fccf291fe5c46708e3c84c1d2d05851 P 43a6e0a26107471fa26598ca375dbdbc [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 14:52:10.923425 18044 raft_consensus.cc:729] T 1fccf291fe5c46708e3c84c1d2d05851 P 43a6e0a26107471fa26598ca375dbdbc [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 43a6e0a26107471fa26598ca375dbdbc, State: Initialized, Role: FOLLOWER
I20260620 14:52:10.923955 18046 raft_consensus.cc:348] T 1fccf291fe5c46708e3c84c1d2d05851 P 85e2888b517740eda27f2796c008948b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85e2888b517740eda27f2796c008948b" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 32787 } } peers { permanent_uuid: "b79dd7c833e24143a22520483886771d" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 36027 } } peers { permanent_uuid: "43a6e0a26107471fa26598ca375dbdbc" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 41611 } }
I20260620 14:52:10.924185 18044 consensus_queue.cc:260] T 1fccf291fe5c46708e3c84c1d2d05851 P 43a6e0a26107471fa26598ca375dbdbc [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85e2888b517740eda27f2796c008948b" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 32787 } } peers { permanent_uuid: "b79dd7c833e24143a22520483886771d" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 36027 } } peers { permanent_uuid: "43a6e0a26107471fa26598ca375dbdbc" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 41611 } }
I20260620 14:52:10.924437 18046 raft_consensus.cc:374] T 1fccf291fe5c46708e3c84c1d2d05851 P 85e2888b517740eda27f2796c008948b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 14:52:10.924542 18046 raft_consensus.cc:729] T 1fccf291fe5c46708e3c84c1d2d05851 P 85e2888b517740eda27f2796c008948b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 85e2888b517740eda27f2796c008948b, State: Initialized, Role: FOLLOWER
I20260620 14:52:10.925271 18046 consensus_queue.cc:260] T 1fccf291fe5c46708e3c84c1d2d05851 P 85e2888b517740eda27f2796c008948b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85e2888b517740eda27f2796c008948b" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 32787 } } peers { permanent_uuid: "b79dd7c833e24143a22520483886771d" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 36027 } } peers { permanent_uuid: "43a6e0a26107471fa26598ca375dbdbc" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 41611 } }
I20260620 14:52:10.925455 17890 heartbeater.cc:503] Master 127.0.97.62:40169 was elected leader, sending a full tablet report...
I20260620 14:52:10.926311 18044 ts_tablet_manager.cc:1428] T 1fccf291fe5c46708e3c84c1d2d05851 P 43a6e0a26107471fa26598ca375dbdbc: Time spent starting tablet: real 0.015s user 0.016s sys 0.000s
I20260620 14:52:10.926440 18024 heartbeater.cc:503] Master 127.0.97.62:40169 was elected leader, sending a full tablet report...
I20260620 14:52:10.927129 18046 ts_tablet_manager.cc:1428] T 1fccf291fe5c46708e3c84c1d2d05851 P 85e2888b517740eda27f2796c008948b: Time spent starting tablet: real 0.014s user 0.007s sys 0.007s
I20260620 14:52:10.930298 18050 raft_consensus.cc:482] T 1fccf291fe5c46708e3c84c1d2d05851 P 43a6e0a26107471fa26598ca375dbdbc [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260620 14:52:10.930619 18050 raft_consensus.cc:504] T 1fccf291fe5c46708e3c84c1d2d05851 P 43a6e0a26107471fa26598ca375dbdbc [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85e2888b517740eda27f2796c008948b" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 32787 } } peers { permanent_uuid: "b79dd7c833e24143a22520483886771d" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 36027 } } peers { permanent_uuid: "43a6e0a26107471fa26598ca375dbdbc" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 41611 } }
I20260620 14:52:10.932631 18050 leader_election.cc:288] T 1fccf291fe5c46708e3c84c1d2d05851 P 43a6e0a26107471fa26598ca375dbdbc [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 85e2888b517740eda27f2796c008948b (127.0.97.3:32787), b79dd7c833e24143a22520483886771d (127.0.97.1:36027)
W20260620 14:52:10.938571 17757 tablet.cc:2369] T 1fccf291fe5c46708e3c84c1d2d05851 P b79dd7c833e24143a22520483886771d: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260620 14:52:10.940018 18038 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:52:10.933594 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.2:46429 (local address 127.0.97.3:32787)
0620 14:52:10.933788 (+ 194us) server_negotiation.cc:206] Beginning negotiation
0620 14:52:10.933796 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0620 14:52:10.934068 (+ 272us) server_negotiation.cc:407] Connection header received
0620 14:52:10.934305 (+ 237us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:52:10.934311 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:52:10.934380 (+ 69us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:52:10.934478 (+ 98us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 14:52:10.935024 (+ 546us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:52:10.936065 (+ 1041us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:52:10.938067 (+ 2002us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:52:10.939031 (+ 964us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:52:10.939187 (+ 156us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:52:10.939301 (+ 114us) server_negotiation.cc:1035] Waiting for connection context
0620 14:52:10.939635 (+ 334us) server_negotiation.cc:299] Negotiation successful
0620 14:52:10.939761 (+ 126us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":44}
I20260620 14:52:10.940196 18052 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:52:10.932985 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.3:32787 (local address 127.0.97.2:46429)
0620 14:52:10.933954 (+ 969us) negotiation.cc:107] Waiting for socket to connect
0620 14:52:10.933974 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0620 14:52:10.934147 (+ 173us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 14:52:10.934492 (+ 345us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 14:52:10.934497 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 14:52:10.934519 (+ 22us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 14:52:10.934869 (+ 350us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:52:10.934883 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:52:10.936194 (+ 1311us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:52:10.936199 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:52:10.937896 (+ 1697us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:52:10.937908 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:52:10.939255 (+ 1347us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:52:10.939260 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:52:10.939411 (+ 151us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:52:10.939436 (+ 25us) client_negotiation.cc:769] Sending connection context
0620 14:52:10.939564 (+ 128us) client_negotiation.cc:240] Negotiation successful
0620 14:52:10.939739 (+ 175us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":578,"mutex_wait_us":23,"thread_start_us":408,"threads_started":1}
I20260620 14:52:10.940676 18040 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:52:10.933576 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.2:47025 (local address 127.0.97.1:36027)
0620 14:52:10.933779 (+ 203us) server_negotiation.cc:206] Beginning negotiation
0620 14:52:10.933786 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0620 14:52:10.934649 (+ 863us) server_negotiation.cc:407] Connection header received
0620 14:52:10.934728 (+ 79us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:52:10.934735 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:52:10.934818 (+ 83us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:52:10.934919 (+ 101us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 14:52:10.935560 (+ 641us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:52:10.936510 (+ 950us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:52:10.938335 (+ 1825us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:52:10.939579 (+ 1244us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:52:10.939798 (+ 219us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:52:10.939905 (+ 107us) server_negotiation.cc:1035] Waiting for connection context
0620 14:52:10.940263 (+ 358us) server_negotiation.cc:299] Negotiation successful
0620 14:52:10.940416 (+ 153us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":51}
I20260620 14:52:10.941457 18053 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:52:10.933041 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:36027 (local address 127.0.97.2:47025)
0620 14:52:10.934141 (+ 1100us) negotiation.cc:107] Waiting for socket to connect
0620 14:52:10.934162 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0620 14:52:10.934283 (+ 121us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 14:52:10.934935 (+ 652us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 14:52:10.934961 (+ 26us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 14:52:10.934988 (+ 27us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 14:52:10.935406 (+ 418us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:52:10.935418 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:52:10.936640 (+ 1222us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:52:10.936644 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:52:10.938144 (+ 1500us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:52:10.938159 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:52:10.939866 (+ 1707us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:52:10.939872 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:52:10.940049 (+ 177us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:52:10.940072 (+ 23us) client_negotiation.cc:769] Sending connection context
0620 14:52:10.940940 (+ 868us) client_negotiation.cc:240] Negotiation successful
0620 14:52:10.941095 (+ 155us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":967,"mutex_wait_us":1,"thread_start_us":746,"threads_started":1}
I20260620 14:52:10.941452 17979 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1fccf291fe5c46708e3c84c1d2d05851" candidate_uuid: "43a6e0a26107471fa26598ca375dbdbc" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "85e2888b517740eda27f2796c008948b" is_pre_election: true
I20260620 14:52:10.942144 17979 raft_consensus.cc:2454] T 1fccf291fe5c46708e3c84c1d2d05851 P 85e2888b517740eda27f2796c008948b [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 43a6e0a26107471fa26598ca375dbdbc in term 0.
I20260620 14:52:10.942698 17711 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1fccf291fe5c46708e3c84c1d2d05851" candidate_uuid: "43a6e0a26107471fa26598ca375dbdbc" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b79dd7c833e24143a22520483886771d" is_pre_election: true
I20260620 14:52:10.943182 17711 raft_consensus.cc:2454] T 1fccf291fe5c46708e3c84c1d2d05851 P b79dd7c833e24143a22520483886771d [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 43a6e0a26107471fa26598ca375dbdbc in term 0.
I20260620 14:52:10.944064 17779 leader_election.cc:302] T 1fccf291fe5c46708e3c84c1d2d05851 P 43a6e0a26107471fa26598ca375dbdbc [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 3 yes votes; 0 no votes. yes voters: 43a6e0a26107471fa26598ca375dbdbc, 85e2888b517740eda27f2796c008948b, b79dd7c833e24143a22520483886771d; no voters:
I20260620 14:52:10.944552 18050 raft_consensus.cc:2781] T 1fccf291fe5c46708e3c84c1d2d05851 P 43a6e0a26107471fa26598ca375dbdbc [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260620 14:52:10.944708 18050 raft_consensus.cc:482] T 1fccf291fe5c46708e3c84c1d2d05851 P 43a6e0a26107471fa26598ca375dbdbc [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260620 14:52:10.944834 18050 raft_consensus.cc:3037] T 1fccf291fe5c46708e3c84c1d2d05851 P 43a6e0a26107471fa26598ca375dbdbc [term 0 FOLLOWER]: Advancing to term 1
I20260620 14:52:10.947027 18050 raft_consensus.cc:504] T 1fccf291fe5c46708e3c84c1d2d05851 P 43a6e0a26107471fa26598ca375dbdbc [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85e2888b517740eda27f2796c008948b" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 32787 } } peers { permanent_uuid: "b79dd7c833e24143a22520483886771d" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 36027 } } peers { permanent_uuid: "43a6e0a26107471fa26598ca375dbdbc" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 41611 } }
I20260620 14:52:10.947772 18050 leader_election.cc:288] T 1fccf291fe5c46708e3c84c1d2d05851 P 43a6e0a26107471fa26598ca375dbdbc [CANDIDATE]: Term 1 election: Requested vote from peers 85e2888b517740eda27f2796c008948b (127.0.97.3:32787), b79dd7c833e24143a22520483886771d (127.0.97.1:36027)
I20260620 14:52:10.948261 17979 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1fccf291fe5c46708e3c84c1d2d05851" candidate_uuid: "43a6e0a26107471fa26598ca375dbdbc" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "85e2888b517740eda27f2796c008948b"
I20260620 14:52:10.948268 17711 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1fccf291fe5c46708e3c84c1d2d05851" candidate_uuid: "43a6e0a26107471fa26598ca375dbdbc" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b79dd7c833e24143a22520483886771d"
I20260620 14:52:10.948496 17979 raft_consensus.cc:3037] T 1fccf291fe5c46708e3c84c1d2d05851 P 85e2888b517740eda27f2796c008948b [term 0 FOLLOWER]: Advancing to term 1
I20260620 14:52:10.948496 17711 raft_consensus.cc:3037] T 1fccf291fe5c46708e3c84c1d2d05851 P b79dd7c833e24143a22520483886771d [term 0 FOLLOWER]: Advancing to term 1
I20260620 14:52:10.950733 17711 raft_consensus.cc:2454] T 1fccf291fe5c46708e3c84c1d2d05851 P b79dd7c833e24143a22520483886771d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 43a6e0a26107471fa26598ca375dbdbc in term 1.
I20260620 14:52:10.950735 17979 raft_consensus.cc:2454] T 1fccf291fe5c46708e3c84c1d2d05851 P 85e2888b517740eda27f2796c008948b [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 43a6e0a26107471fa26598ca375dbdbc in term 1.
I20260620 14:52:10.951206 17779 leader_election.cc:302] T 1fccf291fe5c46708e3c84c1d2d05851 P 43a6e0a26107471fa26598ca375dbdbc [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 43a6e0a26107471fa26598ca375dbdbc, b79dd7c833e24143a22520483886771d; no voters:
I20260620 14:52:10.951568 18050 raft_consensus.cc:2781] T 1fccf291fe5c46708e3c84c1d2d05851 P 43a6e0a26107471fa26598ca375dbdbc [term 1 FOLLOWER]: Leader election won for term 1
I20260620 14:52:10.952065 18050 raft_consensus.cc:686] T 1fccf291fe5c46708e3c84c1d2d05851 P 43a6e0a26107471fa26598ca375dbdbc [term 1 LEADER]: Becoming Leader. State: Replica: 43a6e0a26107471fa26598ca375dbdbc, State: Running, Role: LEADER
I20260620 14:52:10.952566 18050 consensus_queue.cc:237] T 1fccf291fe5c46708e3c84c1d2d05851 P 43a6e0a26107471fa26598ca375dbdbc [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85e2888b517740eda27f2796c008948b" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 32787 } } peers { permanent_uuid: "b79dd7c833e24143a22520483886771d" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 36027 } } peers { permanent_uuid: "43a6e0a26107471fa26598ca375dbdbc" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 41611 } }
I20260620 14:52:10.957613 17567 catalog_manager.cc:5495] T 1fccf291fe5c46708e3c84c1d2d05851 P 43a6e0a26107471fa26598ca375dbdbc reported cstate change: term changed from 0 to 1, leader changed from <none> to 43a6e0a26107471fa26598ca375dbdbc (127.0.97.2). New cstate: current_term: 1 leader_uuid: "43a6e0a26107471fa26598ca375dbdbc" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85e2888b517740eda27f2796c008948b" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 32787 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "b79dd7c833e24143a22520483886771d" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 36027 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "43a6e0a26107471fa26598ca375dbdbc" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 41611 } health_report { overall_health: HEALTHY } } }
W20260620 14:52:10.966544 17891 tablet.cc:2369] T 1fccf291fe5c46708e3c84c1d2d05851 P 43a6e0a26107471fa26598ca375dbdbc: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260620 14:52:10.979854 18041 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:52:10.974215 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56748 (local address 127.0.97.2:41611)
0620 14:52:10.974449 (+ 234us) server_negotiation.cc:206] Beginning negotiation
0620 14:52:10.974454 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 14:52:10.974671 (+ 217us) server_negotiation.cc:407] Connection header received
0620 14:52:10.974830 (+ 159us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:52:10.974834 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:52:10.974923 (+ 89us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:52:10.975057 (+ 134us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 14:52:10.975404 (+ 347us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:52:10.976226 (+ 822us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:52:10.977597 (+ 1371us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:52:10.978010 (+ 413us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:52:10.978104 (+ 94us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:52:10.978618 (+ 514us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 14:52:10.979157 (+ 539us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0620 14:52:10.979331 (+ 174us) server_negotiation.cc:1035] Waiting for connection context
0620 14:52:10.979416 (+ 85us) server_negotiation.cc:299] Negotiation successful
0620 14:52:10.979500 (+ 84us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":91}
I20260620 14:52:10.989315 17979 raft_consensus.cc:1261] T 1fccf291fe5c46708e3c84c1d2d05851 P 85e2888b517740eda27f2796c008948b [term 1 FOLLOWER]: Refusing update from remote peer 43a6e0a26107471fa26598ca375dbdbc: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 14:52:10.989315 17711 raft_consensus.cc:1261] T 1fccf291fe5c46708e3c84c1d2d05851 P b79dd7c833e24143a22520483886771d [term 1 FOLLOWER]: Refusing update from remote peer 43a6e0a26107471fa26598ca375dbdbc: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 14:52:10.990752 18050 consensus_queue.cc:1035] T 1fccf291fe5c46708e3c84c1d2d05851 P 43a6e0a26107471fa26598ca375dbdbc [LEADER]: Connected to new peer: Peer: permanent_uuid: "b79dd7c833e24143a22520483886771d" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 36027 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 14:52:10.991147 18054 consensus_queue.cc:1035] T 1fccf291fe5c46708e3c84c1d2d05851 P 43a6e0a26107471fa26598ca375dbdbc [LEADER]: Connected to new peer: Peer: permanent_uuid: "85e2888b517740eda27f2796c008948b" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 32787 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 14:52:11.007779 18057 mvcc.cc:205] Tried to move back new op lower bound from 7298937368518377472 to 7298937368386002944. Current Snapshot: MvccSnapshot[applied={T|T < 7298937368518377472}]
I20260620 14:52:11.027366 17566 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:32930:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 14:52:11.027763 17566 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:32930:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 14:52:11.032013 17566 catalog_manager.cc:5782] T 00000000000000000000000000000000 P aafe3ca9ef6d450592d1ad1386e36fa4: Sending DeleteTablet for 3 replicas of tablet 1fccf291fe5c46708e3c84c1d2d05851
I20260620 14:52:11.033361 17691 tablet_service.cc:1514] Processing DeleteTablet for tablet 1fccf291fe5c46708e3c84c1d2d05851 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 14:52:11 UTC) from {username='kudu', principal='kudu/127.0.97.62@KRBTEST.COM'} at 127.0.0.1:48046
I20260620 14:52:11.033372 17959 tablet_service.cc:1514] Processing DeleteTablet for tablet 1fccf291fe5c46708e3c84c1d2d05851 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 14:52:11 UTC) from {username='kudu', principal='kudu/127.0.97.62@KRBTEST.COM'} at 127.0.0.1:50330
I20260620 14:52:11.034864 17825 tablet_service.cc:1514] Processing DeleteTablet for tablet 1fccf291fe5c46708e3c84c1d2d05851 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 14:52:11 UTC) from {username='kudu', principal='kudu/127.0.97.62@KRBTEST.COM'} at 127.0.0.1:56740
I20260620 14:52:11.035295 18068 tablet_replica.cc:330] T 1fccf291fe5c46708e3c84c1d2d05851 P 85e2888b517740eda27f2796c008948b: stopping tablet replica
I20260620 14:52:11.035961 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskUci3DU/build/asan/bin/kudu pid 17628
I20260620 14:52:11.037521 18068 raft_consensus.cc:2229] T 1fccf291fe5c46708e3c84c1d2d05851 P 85e2888b517740eda27f2796c008948b [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 14:52:11.038087 18068 raft_consensus.cc:2258] T 1fccf291fe5c46708e3c84c1d2d05851 P 85e2888b517740eda27f2796c008948b [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 14:52:11.039180 18067 tablet_replica.cc:330] T 1fccf291fe5c46708e3c84c1d2d05851 P b79dd7c833e24143a22520483886771d: stopping tablet replica
I20260620 14:52:11.039688 18067 raft_consensus.cc:2229] T 1fccf291fe5c46708e3c84c1d2d05851 P b79dd7c833e24143a22520483886771d [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 14:52:11.040264 18067 raft_consensus.cc:2258] T 1fccf291fe5c46708e3c84c1d2d05851 P b79dd7c833e24143a22520483886771d [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 14:52:11.042423 18068 ts_tablet_manager.cc:1905] T 1fccf291fe5c46708e3c84c1d2d05851 P 85e2888b517740eda27f2796c008948b: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 14:52:11.047622 18069 tablet_replica.cc:330] T 1fccf291fe5c46708e3c84c1d2d05851 P 43a6e0a26107471fa26598ca375dbdbc: stopping tablet replica
I20260620 14:52:11.048262 18069 raft_consensus.cc:2229] T 1fccf291fe5c46708e3c84c1d2d05851 P 43a6e0a26107471fa26598ca375dbdbc [term 1 LEADER]: Raft consensus shutting down.
I20260620 14:52:11.049043 18069 raft_consensus.cc:2258] T 1fccf291fe5c46708e3c84c1d2d05851 P 43a6e0a26107471fa26598ca375dbdbc [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 14:52:11.051738 18069 ts_tablet_manager.cc:1905] T 1fccf291fe5c46708e3c84c1d2d05851 P 43a6e0a26107471fa26598ca375dbdbc: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 14:52:11.053751 18067 ts_tablet_manager.cc:1905] T 1fccf291fe5c46708e3c84c1d2d05851 P b79dd7c833e24143a22520483886771d: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 14:52:11.060143 18069 ts_tablet_manager.cc:1918] T 1fccf291fe5c46708e3c84c1d2d05851 P 43a6e0a26107471fa26598ca375dbdbc: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260620 14:52:11.060346 18069 log.cc:1196] T 1fccf291fe5c46708e3c84c1d2d05851 P 43a6e0a26107471fa26598ca375dbdbc: Deleting WAL directory at /tmp/dist-test-taskUci3DU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781966829091300-388-0/minicluster-data/ts-1/wal/wals/1fccf291fe5c46708e3c84c1d2d05851
I20260620 14:52:11.060921 18069 ts_tablet_manager.cc:1939] T 1fccf291fe5c46708e3c84c1d2d05851 P 43a6e0a26107471fa26598ca375dbdbc: Deleting consensus metadata
I20260620 14:52:11.062951 17553 catalog_manager.cc:4841] TS 43a6e0a26107471fa26598ca375dbdbc (127.0.97.2:41611): tablet 1fccf291fe5c46708e3c84c1d2d05851 (table test-table [id=35790d800f0c4444864a5875107175c3]) successfully deleted
I20260620 14:52:11.064640 18068 ts_tablet_manager.cc:1918] T 1fccf291fe5c46708e3c84c1d2d05851 P 85e2888b517740eda27f2796c008948b: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260620 14:52:11.064824 18068 log.cc:1196] T 1fccf291fe5c46708e3c84c1d2d05851 P 85e2888b517740eda27f2796c008948b: Deleting WAL directory at /tmp/dist-test-taskUci3DU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781966829091300-388-0/minicluster-data/ts-2/wal/wals/1fccf291fe5c46708e3c84c1d2d05851
I20260620 14:52:11.065399 18068 ts_tablet_manager.cc:1939] T 1fccf291fe5c46708e3c84c1d2d05851 P 85e2888b517740eda27f2796c008948b: Deleting consensus metadata
I20260620 14:52:11.063730 18067 ts_tablet_manager.cc:1918] T 1fccf291fe5c46708e3c84c1d2d05851 P b79dd7c833e24143a22520483886771d: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260620 14:52:11.067070 18067 log.cc:1196] T 1fccf291fe5c46708e3c84c1d2d05851 P b79dd7c833e24143a22520483886771d: Deleting WAL directory at /tmp/dist-test-taskUci3DU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781966829091300-388-0/minicluster-data/ts-0/wal/wals/1fccf291fe5c46708e3c84c1d2d05851
I20260620 14:52:11.067446 17553 catalog_manager.cc:4841] TS 85e2888b517740eda27f2796c008948b (127.0.97.3:32787): tablet 1fccf291fe5c46708e3c84c1d2d05851 (table test-table [id=35790d800f0c4444864a5875107175c3]) successfully deleted
I20260620 14:52:11.067716 18067 ts_tablet_manager.cc:1939] T 1fccf291fe5c46708e3c84c1d2d05851 P b79dd7c833e24143a22520483886771d: Deleting consensus metadata
I20260620 14:52:11.069356 17552 catalog_manager.cc:4841] TS b79dd7c833e24143a22520483886771d (127.0.97.1:36027): tablet 1fccf291fe5c46708e3c84c1d2d05851 (table test-table [id=35790d800f0c4444864a5875107175c3]) successfully deleted
Jun 20 14:52:11 dist-test-slave-q9j6 krb5kdc[17522](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781967128, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.1@KRBTEST.COM
I20260620 14:52:11.081442 18040 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:52:11.041014 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48062 (local address 127.0.97.1:36027)
0620 14:52:11.041199 (+ 185us) server_negotiation.cc:206] Beginning negotiation
0620 14:52:11.041206 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0620 14:52:11.052273 (+ 11067us) server_negotiation.cc:407] Connection header received
0620 14:52:11.056836 (+ 4563us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:52:11.056843 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:52:11.056937 (+ 94us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:52:11.057024 (+ 87us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:52:11.063804 (+ 6780us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:52:11.064928 (+ 1124us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:52:11.069271 (+ 4343us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:52:11.069858 (+ 587us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:52:11.069946 (+ 88us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:52:11.074786 (+ 4840us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 14:52:11.074805 (+ 19us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 14:52:11.074823 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 14:52:11.074869 (+ 46us) server_negotiation.cc:922] calling sasl_server_start()
0620 14:52:11.078008 (+ 3139us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:52:11.078893 (+ 885us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:52:11.078901 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:52:11.078903 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:52:11.079056 (+ 153us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:52:11.079566 (+ 510us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:52:11.079572 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:52:11.079575 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:52:11.080055 (+ 480us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 14:52:11.080285 (+ 230us) server_negotiation.cc:1035] Waiting for connection context
0620 14:52:11.080810 (+ 525us) server_negotiation.cc:299] Negotiation successful
0620 14:52:11.081056 (+ 246us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":62}
I20260620 14:52:11.082130 17751 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 14:52:11.189242 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskUci3DU/build/asan/bin/kudu with pid 17628
I20260620 14:52:11.207852 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskUci3DU/build/asan/bin/kudu pid 17762
Jun 20 14:52:11 dist-test-slave-q9j6 krb5kdc[17522](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781967128, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.2@KRBTEST.COM
I20260620 14:52:11.228907 18041 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:52:11.208816 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56758 (local address 127.0.97.2:41611)
0620 14:52:11.209039 (+ 223us) server_negotiation.cc:206] Beginning negotiation
0620 14:52:11.209046 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0620 14:52:11.209074 (+ 28us) server_negotiation.cc:407] Connection header received
0620 14:52:11.209147 (+ 73us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:52:11.209153 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:52:11.209235 (+ 82us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:52:11.209352 (+ 117us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:52:11.210603 (+ 1251us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:52:11.212120 (+ 1517us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:52:11.214091 (+ 1971us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:52:11.214832 (+ 741us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:52:11.214988 (+ 156us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:52:11.220847 (+ 5859us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 14:52:11.220863 (+ 16us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 14:52:11.220883 (+ 20us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 14:52:11.220940 (+ 57us) server_negotiation.cc:922] calling sasl_server_start()
0620 14:52:11.225290 (+ 4350us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:52:11.226390 (+ 1100us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:52:11.226403 (+ 13us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:52:11.226407 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:52:11.226611 (+ 204us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:52:11.227075 (+ 464us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:52:11.227080 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:52:11.227083 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:52:11.227576 (+ 493us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 14:52:11.227951 (+ 375us) server_negotiation.cc:1035] Waiting for connection context
0620 14:52:11.228284 (+ 333us) server_negotiation.cc:299] Negotiation successful
0620 14:52:11.228527 (+ 243us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":72}
I20260620 14:52:11.229714 17885 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 14:52:11.346117 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskUci3DU/build/asan/bin/kudu with pid 17762
I20260620 14:52:11.362483 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskUci3DU/build/asan/bin/kudu pid 17896
Jun 20 14:52:11 dist-test-slave-q9j6 krb5kdc[17522](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781967128, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.3@KRBTEST.COM
I20260620 14:52:11.378257 18038 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:52:11.363285 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50336 (local address 127.0.97.3:32787)
0620 14:52:11.363448 (+ 163us) server_negotiation.cc:206] Beginning negotiation
0620 14:52:11.363453 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 14:52:11.363493 (+ 40us) server_negotiation.cc:407] Connection header received
0620 14:52:11.363725 (+ 232us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:52:11.363729 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:52:11.363797 (+ 68us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:52:11.363886 (+ 89us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:52:11.365216 (+ 1330us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:52:11.365944 (+ 728us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:52:11.367197 (+ 1253us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:52:11.367737 (+ 540us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:52:11.367831 (+ 94us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:52:11.372355 (+ 4524us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 14:52:11.372378 (+ 23us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 14:52:11.372399 (+ 21us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 14:52:11.372447 (+ 48us) server_negotiation.cc:922] calling sasl_server_start()
0620 14:52:11.375479 (+ 3032us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:52:11.376382 (+ 903us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:52:11.376391 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:52:11.376394 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:52:11.376548 (+ 154us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:52:11.376920 (+ 372us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:52:11.376923 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:52:11.376925 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:52:11.377288 (+ 363us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 14:52:11.377505 (+ 217us) server_negotiation.cc:1035] Waiting for connection context
0620 14:52:11.377812 (+ 307us) server_negotiation.cc:299] Negotiation successful
0620 14:52:11.377996 (+ 184us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":50}
I20260620 14:52:11.378968 18019 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 14:52:11.470979 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskUci3DU/build/asan/bin/kudu with pid 17896
I20260620 14:52:11.488467 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskUci3DU/build/asan/bin/kudu pid 17535
I20260620 14:52:11.489724 17597 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 14:52:11.604852 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskUci3DU/build/asan/bin/kudu with pid 17535
2026-06-20T14:52:11Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/0 (4394 ms)
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/1
2026-06-20T14:52:11Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T14:52:11Z Disabled control of system clock
I20260620 14:52:11.684023 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskUci3DU/build/asan/bin/kudu
/tmp/dist-test-taskUci3DU/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskUci3DU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781966829091300-388-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskUci3DU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781966829091300-388-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskUci3DU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781966829091300-388-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskUci3DU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781966829091300-388-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:45341
--webserver_interface=127.0.97.62
--webserver_port=0
--builtin_ntp_servers=127.0.97.20:43479
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.97.62:45341
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260620 14:52:12.035327 18078 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 14:52:12.035759 18078 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 14:52:12.035930 18078 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 14:52:12.045411 18078 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 14:52:12.045543 18078 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 14:52:12.045640 18078 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 14:52:12.045722 18078 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 14:52:12.045805 18078 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 14:52:12.055994 18078 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:43479
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskUci3DU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781966829091300-388-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskUci3DU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781966829091300-388-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.97.62:45341
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:45341
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskUci3DU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781966829091300-388-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.97.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskUci3DU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781966829091300-388-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 13:41:20 UTC on ccycloud.quasar-vdagoi.root.comops.site
build id 4457462
ASAN enabled
I20260620 14:52:12.057474 18078 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 14:52:12.059580 18078 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 14:52:12.073097 18084 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 14:52:12.073190 18083 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 14:52:12.073123 18086 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 14:52:12.074550 18078 server_base.cc:1034] running on GCE node
I20260620 14:52:12.075534 18078 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 14:52:12.077364 18078 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 14:52:12.078619 18078 hybrid_clock.cc:648] HybridClock initialized: now 1781967132078565 us; error 49 us; skew 500 ppm
I20260620 14:52:12.079089 18078 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260620 14:52:12.085547 18078 webserver.cc:460] Webserver started at http://127.0.97.62:39259/ using document root <none> and password file <none>
I20260620 14:52:12.086345 18078 fs_manager.cc:359] Metadata directory not provided
I20260620 14:52:12.086447 18078 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 14:52:12.086745 18078 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 14:52:12.090159 18078 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskUci3DU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781966829091300-388-0/minicluster-data/master-0/data/instance:
uuid: "acb0b0114bf342d59b50f175139edc47"
format_stamp: "Formatted at 2026-06-20 14:52:12 on dist-test-slave-q9j6"
server_key: "fc8155f490b29f7c096d0787be4f2695"
server_key_iv: "c1997d597a0ee1a9c86c867d19d0726e"
server_key_version: "encryptionkey@0"
I20260620 14:52:12.091012 18078 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskUci3DU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781966829091300-388-0/minicluster-data/master-0/wal/instance:
uuid: "acb0b0114bf342d59b50f175139edc47"
format_stamp: "Formatted at 2026-06-20 14:52:12 on dist-test-slave-q9j6"
server_key: "fc8155f490b29f7c096d0787be4f2695"
server_key_iv: "c1997d597a0ee1a9c86c867d19d0726e"
server_key_version: "encryptionkey@0"
I20260620 14:52:12.096969 18078 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.007s sys 0.001s
I20260620 14:52:12.101362 18092 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 14:52:12.103111 18078 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.005s sys 0.000s
I20260620 14:52:12.103363 18078 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskUci3DU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781966829091300-388-0/minicluster-data/master-0/data,/tmp/dist-test-taskUci3DU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781966829091300-388-0/minicluster-data/master-0/wal
uuid: "acb0b0114bf342d59b50f175139edc47"
format_stamp: "Formatted at 2026-06-20 14:52:12 on dist-test-slave-q9j6"
server_key: "fc8155f490b29f7c096d0787be4f2695"
server_key_iv: "c1997d597a0ee1a9c86c867d19d0726e"
server_key_version: "encryptionkey@0"
I20260620 14:52:12.103713 18078 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskUci3DU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781966829091300-388-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskUci3DU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781966829091300-388-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskUci3DU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781966829091300-388-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 14:52:12.120812 18078 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 14:52:12.121803 18078 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 14:52:12.122154 18078 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 14:52:12.143409 18078 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.62:45341
I20260620 14:52:12.143429 18143 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.62:45341 every 8 connection(s)
I20260620 14:52:12.145097 18078 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskUci3DU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781966829091300-388-0/minicluster-data/master-0/data/info.pb
I20260620 14:52:12.149297 18144 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 14:52:12.150681 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskUci3DU/build/asan/bin/kudu as pid 18078
I20260620 14:52:12.150876 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskUci3DU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781966829091300-388-0/minicluster-data/master-0/wal/instance
I20260620 14:52:12.151485 388 external_mini_cluster.cc:1428] Setting key d6ab7fdeba98b55623472dad94650cbf
I20260620 14:52:12.163528 18144 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P acb0b0114bf342d59b50f175139edc47: Bootstrap starting.
I20260620 14:52:12.166204 18147 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:52:12.153751 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39322 (local address 127.0.97.62:45341)
0620 14:52:12.154919 (+ 1168us) server_negotiation.cc:206] Beginning negotiation
0620 14:52:12.154932 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0620 14:52:12.154981 (+ 49us) server_negotiation.cc:407] Connection header received
0620 14:52:12.156773 (+ 1792us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:52:12.156804 (+ 31us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:52:12.157384 (+ 580us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:52:12.157846 (+ 462us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:52:12.158465 (+ 619us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:52:12.159845 (+ 1380us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:52:12.161340 (+ 1495us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:52:12.162110 (+ 770us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:52:12.162347 (+ 237us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:52:12.163685 (+ 1338us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 14:52:12.163696 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 14:52:12.163719 (+ 23us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 14:52:12.164341 (+ 622us) server_negotiation.cc:922] calling sasl_server_start()
0620 14:52:12.164428 (+ 87us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 14:52:12.164452 (+ 24us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 14:52:12.164584 (+ 132us) server_negotiation.cc:1035] Waiting for connection context
0620 14:52:12.164826 (+ 242us) server_negotiation.cc:299] Negotiation successful
0620 14:52:12.165185 (+ 359us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":676,"thread_start_us":404,"threads_started":1}
I20260620 14:52:12.167174 18144 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P acb0b0114bf342d59b50f175139edc47: Neither blocks nor log segments found. Creating new log.
I20260620 14:52:12.168329 18144 log.cc:824] T 00000000000000000000000000000000 P acb0b0114bf342d59b50f175139edc47: Log is configured to *not* fsync() on all Append() calls
I20260620 14:52:12.171267 18144 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P acb0b0114bf342d59b50f175139edc47: No bootstrap required, opened a new log
I20260620 14:52:12.178905 18144 raft_consensus.cc:348] T 00000000000000000000000000000000 P acb0b0114bf342d59b50f175139edc47 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "acb0b0114bf342d59b50f175139edc47" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 45341 } }
I20260620 14:52:12.179389 18144 raft_consensus.cc:374] T 00000000000000000000000000000000 P acb0b0114bf342d59b50f175139edc47 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 14:52:12.179505 18144 raft_consensus.cc:729] T 00000000000000000000000000000000 P acb0b0114bf342d59b50f175139edc47 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: acb0b0114bf342d59b50f175139edc47, State: Initialized, Role: FOLLOWER
I20260620 14:52:12.180128 18144 consensus_queue.cc:260] T 00000000000000000000000000000000 P acb0b0114bf342d59b50f175139edc47 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "acb0b0114bf342d59b50f175139edc47" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 45341 } }
I20260620 14:52:12.180383 18144 raft_consensus.cc:388] T 00000000000000000000000000000000 P acb0b0114bf342d59b50f175139edc47 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 14:52:12.180532 18144 raft_consensus.cc:482] T 00000000000000000000000000000000 P acb0b0114bf342d59b50f175139edc47 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 14:52:12.180706 18144 raft_consensus.cc:3037] T 00000000000000000000000000000000 P acb0b0114bf342d59b50f175139edc47 [term 0 FOLLOWER]: Advancing to term 1
I20260620 14:52:12.183109 18144 raft_consensus.cc:504] T 00000000000000000000000000000000 P acb0b0114bf342d59b50f175139edc47 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "acb0b0114bf342d59b50f175139edc47" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 45341 } }
I20260620 14:52:12.183847 18144 leader_election.cc:302] T 00000000000000000000000000000000 P acb0b0114bf342d59b50f175139edc47 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: acb0b0114bf342d59b50f175139edc47; no voters:
I20260620 14:52:12.184448 18144 leader_election.cc:288] T 00000000000000000000000000000000 P acb0b0114bf342d59b50f175139edc47 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260620 14:52:12.184619 18149 raft_consensus.cc:2781] T 00000000000000000000000000000000 P acb0b0114bf342d59b50f175139edc47 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 14:52:12.185376 18149 raft_consensus.cc:686] T 00000000000000000000000000000000 P acb0b0114bf342d59b50f175139edc47 [term 1 LEADER]: Becoming Leader. State: Replica: acb0b0114bf342d59b50f175139edc47, State: Running, Role: LEADER
I20260620 14:52:12.185971 18149 consensus_queue.cc:237] T 00000000000000000000000000000000 P acb0b0114bf342d59b50f175139edc47 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "acb0b0114bf342d59b50f175139edc47" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 45341 } }
I20260620 14:52:12.186800 18144 sys_catalog.cc:564] T 00000000000000000000000000000000 P acb0b0114bf342d59b50f175139edc47 [sys.catalog]: configured and running, proceeding with master startup.
I20260620 14:52:12.188743 18151 sys_catalog.cc:455] T 00000000000000000000000000000000 P acb0b0114bf342d59b50f175139edc47 [sys.catalog]: SysCatalogTable state changed. Reason: New leader acb0b0114bf342d59b50f175139edc47. Latest consensus state: current_term: 1 leader_uuid: "acb0b0114bf342d59b50f175139edc47" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "acb0b0114bf342d59b50f175139edc47" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 45341 } } }
I20260620 14:52:12.189193 18151 sys_catalog.cc:458] T 00000000000000000000000000000000 P acb0b0114bf342d59b50f175139edc47 [sys.catalog]: This master's current role is: LEADER
I20260620 14:52:12.189287 18150 sys_catalog.cc:455] T 00000000000000000000000000000000 P acb0b0114bf342d59b50f175139edc47 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "acb0b0114bf342d59b50f175139edc47" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "acb0b0114bf342d59b50f175139edc47" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 45341 } } }
I20260620 14:52:12.189589 18150 sys_catalog.cc:458] T 00000000000000000000000000000000 P acb0b0114bf342d59b50f175139edc47 [sys.catalog]: This master's current role is: LEADER
I20260620 14:52:12.192314 18154 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 14:52:12.198935 18154 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 14:52:12.209703 18154 catalog_manager.cc:1347] Generated new cluster ID: c5a055941d1a4745976cb95246969fcf
I20260620 14:52:12.209852 18154 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 14:52:12.240964 18154 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 14:52:12.242992 18154 catalog_manager.cc:1504] Loading token signing keys...
I20260620 14:52:12.253187 18154 catalog_manager.cc:5868] T 00000000000000000000000000000000 P acb0b0114bf342d59b50f175139edc47: Generated new TSK 0
I20260620 14:52:12.254196 18154 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260620 14:52:12.258662 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskUci3DU/build/asan/bin/kudu
/tmp/dist-test-taskUci3DU/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskUci3DU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781966829091300-388-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskUci3DU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781966829091300-388-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskUci3DU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781966829091300-388-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskUci3DU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781966829091300-388-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.1:0
--local_ip_for_outbound_sockets=127.0.97.1
--webserver_interface=127.0.97.1
--webserver_port=0
--tserver_master_addrs=127.0.97.62:45341
--builtin_ntp_servers=127.0.97.20:43479
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260620 14:52:12.638351 18168 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 14:52:12.638667 18168 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 14:52:12.638760 18168 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 14:52:12.646548 18168 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 14:52:12.646682 18168 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 14:52:12.646795 18168 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.1
I20260620 14:52:12.656740 18168 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:43479
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskUci3DU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781966829091300-388-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskUci3DU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781966829091300-388-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskUci3DU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781966829091300-388-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.97.1
--webserver_port=0
--tserver_master_addrs=127.0.97.62:45341
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.1
--log_dir=/tmp/dist-test-taskUci3DU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781966829091300-388-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 13:41:20 UTC on ccycloud.quasar-vdagoi.root.comops.site
build id 4457462
ASAN enabled
I20260620 14:52:12.658106 18168 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 14:52:12.660007 18168 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 14:52:12.674870 18173 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 14:52:12.674865 18174 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 14:52:12.674943 18176 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 14:52:12.676141 18168 server_base.cc:1034] running on GCE node
I20260620 14:52:12.676841 18168 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 14:52:12.677910 18168 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 14:52:12.679181 18168 hybrid_clock.cc:648] HybridClock initialized: now 1781967132679109 us; error 72 us; skew 500 ppm
I20260620 14:52:12.679553 18168 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260620 14:52:12.682041 18168 webserver.cc:460] Webserver started at http://127.0.97.1:44995/ using document root <none> and password file <none>
I20260620 14:52:12.682826 18168 fs_manager.cc:359] Metadata directory not provided
I20260620 14:52:12.682971 18168 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 14:52:12.683332 18168 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 14:52:12.686668 18168 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskUci3DU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781966829091300-388-0/minicluster-data/ts-0/data/instance:
uuid: "6a51540ede984c80a500084f8c4e009f"
format_stamp: "Formatted at 2026-06-20 14:52:12 on dist-test-slave-q9j6"
server_key: "3b87324d89a5c71e05e8542b17085fa8"
server_key_iv: "6499f82e88377cb0e53927bebcd8f667"
server_key_version: "encryptionkey@0"
I20260620 14:52:12.687668 18168 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskUci3DU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781966829091300-388-0/minicluster-data/ts-0/wal/instance:
uuid: "6a51540ede984c80a500084f8c4e009f"
format_stamp: "Formatted at 2026-06-20 14:52:12 on dist-test-slave-q9j6"
server_key: "3b87324d89a5c71e05e8542b17085fa8"
server_key_iv: "6499f82e88377cb0e53927bebcd8f667"
server_key_version: "encryptionkey@0"
I20260620 14:52:12.693411 18168 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.004s sys 0.001s
I20260620 14:52:12.697379 18182 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 14:52:12.698912 18168 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260620 14:52:12.699124 18168 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskUci3DU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781966829091300-388-0/minicluster-data/ts-0/data,/tmp/dist-test-taskUci3DU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781966829091300-388-0/minicluster-data/ts-0/wal
uuid: "6a51540ede984c80a500084f8c4e009f"
format_stamp: "Formatted at 2026-06-20 14:52:12 on dist-test-slave-q9j6"
server_key: "3b87324d89a5c71e05e8542b17085fa8"
server_key_iv: "6499f82e88377cb0e53927bebcd8f667"
server_key_version: "encryptionkey@0"
I20260620 14:52:12.699378 18168 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskUci3DU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781966829091300-388-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskUci3DU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781966829091300-388-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskUci3DU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781966829091300-388-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 14:52:12.714553 18168 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 14:52:12.715492 18168 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 14:52:12.715832 18168 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 14:52:12.716874 18168 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 14:52:12.718468 18168 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 14:52:12.718592 18168 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 14:52:12.718740 18168 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 14:52:12.718830 18168 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 14:52:12.752622 18168 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.1:38009
I20260620 14:52:12.752710 18294 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.1:38009 every 8 connection(s)
I20260620 14:52:12.754421 18168 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskUci3DU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781966829091300-388-0/minicluster-data/ts-0/data/info.pb
I20260620 14:52:12.759443 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskUci3DU/build/asan/bin/kudu as pid 18168
I20260620 14:52:12.759661 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskUci3DU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781966829091300-388-0/minicluster-data/ts-0/wal/instance
I20260620 14:52:12.760125 388 external_mini_cluster.cc:1428] Setting key 11ad1867a38fed342fc27e013d227582
I20260620 14:52:12.762858 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskUci3DU/build/asan/bin/kudu
/tmp/dist-test-taskUci3DU/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskUci3DU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781966829091300-388-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskUci3DU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781966829091300-388-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskUci3DU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781966829091300-388-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskUci3DU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781966829091300-388-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.2:0
--local_ip_for_outbound_sockets=127.0.97.2
--webserver_interface=127.0.97.2
--webserver_port=0
--tserver_master_addrs=127.0.97.62:45341
--builtin_ntp_servers=127.0.97.20:43479
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
I20260620 14:52:12.767941 18297 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:52:12.757158 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.1:49049 (local address 127.0.97.62:45341)
0620 14:52:12.757638 (+ 480us) server_negotiation.cc:206] Beginning negotiation
0620 14:52:12.757645 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0620 14:52:12.758951 (+ 1306us) server_negotiation.cc:407] Connection header received
0620 14:52:12.760989 (+ 2038us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:52:12.760995 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:52:12.761084 (+ 89us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:52:12.761221 (+ 137us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:52:12.762389 (+ 1168us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:52:12.763218 (+ 829us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:52:12.764917 (+ 1699us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:52:12.765352 (+ 435us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:52:12.765504 (+ 152us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:52:12.767020 (+ 1516us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 14:52:12.767031 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 14:52:12.767034 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 14:52:12.767058 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0620 14:52:12.767100 (+ 42us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 14:52:12.767114 (+ 14us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 14:52:12.767213 (+ 99us) server_negotiation.cc:1035] Waiting for connection context
0620 14:52:12.767450 (+ 237us) server_negotiation.cc:299] Negotiation successful
0620 14:52:12.767556 (+ 106us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":356,"thread_start_us":252,"threads_started":1}
I20260620 14:52:12.769066 18298 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:52:12.757825 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:45341 (local address 127.0.97.1:49049)
0620 14:52:12.758778 (+ 953us) negotiation.cc:107] Waiting for socket to connect
0620 14:52:12.758816 (+ 38us) client_negotiation.cc:174] Beginning negotiation
0620 14:52:12.760696 (+ 1880us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 14:52:12.761481 (+ 785us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 14:52:12.761491 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 14:52:12.761610 (+ 119us) client_negotiation.cc:189] Negotiated authn=SASL
0620 14:52:12.762212 (+ 602us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:52:12.762230 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:52:12.763387 (+ 1157us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:52:12.763391 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:52:12.764759 (+ 1368us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:52:12.764772 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:52:12.765605 (+ 833us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:52:12.765612 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:52:12.765876 (+ 264us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:52:12.766655 (+ 779us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0620 14:52:12.766658 (+ 3us) client_negotiation.cc:656] calling sasl_client_start()
0620 14:52:12.766702 (+ 44us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0620 14:52:12.766801 (+ 99us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 14:52:12.767214 (+ 413us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 14:52:12.767228 (+ 14us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 14:52:12.767239 (+ 11us) client_negotiation.cc:769] Sending connection context
0620 14:52:12.767442 (+ 203us) client_negotiation.cc:240] Negotiation successful
0620 14:52:12.767916 (+ 474us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":488,"thread_start_us":310,"threads_started":1}
I20260620 14:52:12.770871 18295 heartbeater.cc:347] Connected to a master server at 127.0.97.62:45341
I20260620 14:52:12.771212 18295 heartbeater.cc:464] Registering TS with master...
I20260620 14:52:12.772228 18295 heartbeater.cc:511] Master 127.0.97.62:45341 requested a full tablet report, sending...
I20260620 14:52:12.774894 18109 ts_manager.cc:194] Registered new tserver with Master: 6a51540ede984c80a500084f8c4e009f (127.0.97.1:38009)
I20260620 14:52:12.777179 18109 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.97.1:49049
W20260620 14:52:13.114019 18299 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 14:52:13.114341 18299 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 14:52:13.114428 18299 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 14:52:13.121948 18299 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 14:52:13.122074 18299 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 14:52:13.122187 18299 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.2
I20260620 14:52:13.131861 18299 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:43479
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskUci3DU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781966829091300-388-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskUci3DU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781966829091300-388-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskUci3DU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781966829091300-388-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.97.2
--webserver_port=0
--tserver_master_addrs=127.0.97.62:45341
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.2
--log_dir=/tmp/dist-test-taskUci3DU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781966829091300-388-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 13:41:20 UTC on ccycloud.quasar-vdagoi.root.comops.site
build id 4457462
ASAN enabled
I20260620 14:52:13.133208 18299 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 14:52:13.135082 18299 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 14:52:13.149197 18305 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 14:52:13.149825 18304 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 14:52:13.149937 18307 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 14:52:13.150429 18299 server_base.cc:1034] running on GCE node
I20260620 14:52:13.151134 18299 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 14:52:13.152374 18299 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 14:52:13.153662 18299 hybrid_clock.cc:648] HybridClock initialized: now 1781967133153605 us; error 52 us; skew 500 ppm
I20260620 14:52:13.154019 18299 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260620 14:52:13.156765 18299 webserver.cc:460] Webserver started at http://127.0.97.2:33979/ using document root <none> and password file <none>
I20260620 14:52:13.157567 18299 fs_manager.cc:359] Metadata directory not provided
I20260620 14:52:13.157711 18299 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 14:52:13.158070 18299 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 14:52:13.161377 18299 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskUci3DU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781966829091300-388-0/minicluster-data/ts-1/data/instance:
uuid: "8de773159a86446ea4716aa302c0e6ed"
format_stamp: "Formatted at 2026-06-20 14:52:13 on dist-test-slave-q9j6"
server_key: "09f0539439a62b0197662b57f14bffcd"
server_key_iv: "32d414119bfa321bb0ef60657a3bbbac"
server_key_version: "encryptionkey@0"
I20260620 14:52:13.162298 18299 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskUci3DU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781966829091300-388-0/minicluster-data/ts-1/wal/instance:
uuid: "8de773159a86446ea4716aa302c0e6ed"
format_stamp: "Formatted at 2026-06-20 14:52:13 on dist-test-slave-q9j6"
server_key: "09f0539439a62b0197662b57f14bffcd"
server_key_iv: "32d414119bfa321bb0ef60657a3bbbac"
server_key_version: "encryptionkey@0"
I20260620 14:52:13.168071 18299 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.003s sys 0.005s
I20260620 14:52:13.172073 18313 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 14:52:13.173535 18299 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.001s
I20260620 14:52:13.173779 18299 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskUci3DU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781966829091300-388-0/minicluster-data/ts-1/data,/tmp/dist-test-taskUci3DU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781966829091300-388-0/minicluster-data/ts-1/wal
uuid: "8de773159a86446ea4716aa302c0e6ed"
format_stamp: "Formatted at 2026-06-20 14:52:13 on dist-test-slave-q9j6"
server_key: "09f0539439a62b0197662b57f14bffcd"
server_key_iv: "32d414119bfa321bb0ef60657a3bbbac"
server_key_version: "encryptionkey@0"
I20260620 14:52:13.174049 18299 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskUci3DU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781966829091300-388-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskUci3DU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781966829091300-388-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskUci3DU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781966829091300-388-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 14:52:13.189399 18299 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 14:52:13.190347 18299 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 14:52:13.190661 18299 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 14:52:13.191823 18299 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 14:52:13.193656 18299 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 14:52:13.193800 18299 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 14:52:13.193934 18299 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 14:52:13.194024 18299 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 14:52:13.230397 18299 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.2:38417
I20260620 14:52:13.230454 18425 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.2:38417 every 8 connection(s)
I20260620 14:52:13.231968 18299 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskUci3DU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781966829091300-388-0/minicluster-data/ts-1/data/info.pb
I20260620 14:52:13.232462 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskUci3DU/build/asan/bin/kudu as pid 18299
I20260620 14:52:13.232658 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskUci3DU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781966829091300-388-0/minicluster-data/ts-1/wal/instance
I20260620 14:52:13.233307 388 external_mini_cluster.cc:1428] Setting key 23da79be138c012bbd4c017ddb61d5e7
I20260620 14:52:13.236977 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskUci3DU/build/asan/bin/kudu
/tmp/dist-test-taskUci3DU/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskUci3DU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781966829091300-388-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskUci3DU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781966829091300-388-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskUci3DU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781966829091300-388-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskUci3DU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781966829091300-388-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.3:0
--local_ip_for_outbound_sockets=127.0.97.3
--webserver_interface=127.0.97.3
--webserver_port=0
--tserver_master_addrs=127.0.97.62:45341
--builtin_ntp_servers=127.0.97.20:43479
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
I20260620 14:52:13.249938 18297 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:52:13.235267 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.2:36083 (local address 127.0.97.62:45341)
0620 14:52:13.235476 (+ 209us) server_negotiation.cc:206] Beginning negotiation
0620 14:52:13.235484 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0620 14:52:13.237475 (+ 1991us) server_negotiation.cc:407] Connection header received
0620 14:52:13.240637 (+ 3162us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:52:13.240642 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:52:13.240722 (+ 80us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:52:13.240812 (+ 90us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:52:13.242530 (+ 1718us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:52:13.243278 (+ 748us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:52:13.246153 (+ 2875us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:52:13.247034 (+ 881us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:52:13.247207 (+ 173us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:52:13.249129 (+ 1922us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 14:52:13.249134 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 14:52:13.249138 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 14:52:13.249160 (+ 22us) server_negotiation.cc:922] calling sasl_server_start()
0620 14:52:13.249198 (+ 38us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 14:52:13.249211 (+ 13us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 14:52:13.249295 (+ 84us) server_negotiation.cc:1035] Waiting for connection context
0620 14:52:13.249587 (+ 292us) server_negotiation.cc:299] Negotiation successful
0620 14:52:13.249684 (+ 97us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":64}
I20260620 14:52:13.251708 18428 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:52:13.236038 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:45341 (local address 127.0.97.2:36083)
0620 14:52:13.237290 (+ 1252us) negotiation.cc:107] Waiting for socket to connect
0620 14:52:13.237343 (+ 53us) client_negotiation.cc:174] Beginning negotiation
0620 14:52:13.240237 (+ 2894us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 14:52:13.241146 (+ 909us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 14:52:13.241162 (+ 16us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 14:52:13.241329 (+ 167us) client_negotiation.cc:189] Negotiated authn=SASL
0620 14:52:13.242285 (+ 956us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:52:13.242317 (+ 32us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:52:13.243478 (+ 1161us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:52:13.243487 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:52:13.245889 (+ 2402us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:52:13.245912 (+ 23us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:52:13.247325 (+ 1413us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:52:13.247333 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:52:13.247797 (+ 464us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:52:13.248654 (+ 857us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0620 14:52:13.248660 (+ 6us) client_negotiation.cc:656] calling sasl_client_start()
0620 14:52:13.248725 (+ 65us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0620 14:52:13.248870 (+ 145us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 14:52:13.249322 (+ 452us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 14:52:13.249336 (+ 14us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 14:52:13.249347 (+ 11us) client_negotiation.cc:769] Sending connection context
0620 14:52:13.249554 (+ 207us) client_negotiation.cc:240] Negotiation successful
0620 14:52:13.250078 (+ 524us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":597,"thread_start_us":360,"threads_started":1}
I20260620 14:52:13.253480 18426 heartbeater.cc:347] Connected to a master server at 127.0.97.62:45341
I20260620 14:52:13.253859 18426 heartbeater.cc:464] Registering TS with master...
I20260620 14:52:13.254911 18426 heartbeater.cc:511] Master 127.0.97.62:45341 requested a full tablet report, sending...
I20260620 14:52:13.256773 18109 ts_manager.cc:194] Registered new tserver with Master: 8de773159a86446ea4716aa302c0e6ed (127.0.97.2:38417)
I20260620 14:52:13.257642 18109 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.97.2:36083
W20260620 14:52:13.585299 18429 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 14:52:13.585719 18429 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 14:52:13.585829 18429 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 14:52:13.593524 18429 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 14:52:13.593649 18429 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 14:52:13.593760 18429 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.3
I20260620 14:52:13.603322 18429 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:43479
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskUci3DU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781966829091300-388-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskUci3DU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781966829091300-388-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskUci3DU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781966829091300-388-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.97.3
--webserver_port=0
--tserver_master_addrs=127.0.97.62:45341
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.3
--log_dir=/tmp/dist-test-taskUci3DU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781966829091300-388-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 13:41:20 UTC on ccycloud.quasar-vdagoi.root.comops.site
build id 4457462
ASAN enabled
I20260620 14:52:13.604943 18429 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 14:52:13.606848 18429 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 14:52:13.621732 18434 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 14:52:13.621645 18437 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 14:52:13.621644 18435 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 14:52:13.622718 18429 server_base.cc:1034] running on GCE node
I20260620 14:52:13.623448 18429 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 14:52:13.624614 18429 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 14:52:13.625880 18429 hybrid_clock.cc:648] HybridClock initialized: now 1781967133625827 us; error 51 us; skew 500 ppm
I20260620 14:52:13.626255 18429 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260620 14:52:13.628835 18429 webserver.cc:460] Webserver started at http://127.0.97.3:39117/ using document root <none> and password file <none>
I20260620 14:52:13.629657 18429 fs_manager.cc:359] Metadata directory not provided
I20260620 14:52:13.629814 18429 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 14:52:13.630172 18429 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 14:52:13.633466 18429 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskUci3DU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781966829091300-388-0/minicluster-data/ts-2/data/instance:
uuid: "eae4e91c1b6e43aea4d1ba790802a0d9"
format_stamp: "Formatted at 2026-06-20 14:52:13 on dist-test-slave-q9j6"
server_key: "c31aacc2ff2c0dccfef02ea5cc9a0ca6"
server_key_iv: "8617378c2c89d61478935247bb3f6d70"
server_key_version: "encryptionkey@0"
I20260620 14:52:13.634375 18429 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskUci3DU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781966829091300-388-0/minicluster-data/ts-2/wal/instance:
uuid: "eae4e91c1b6e43aea4d1ba790802a0d9"
format_stamp: "Formatted at 2026-06-20 14:52:13 on dist-test-slave-q9j6"
server_key: "c31aacc2ff2c0dccfef02ea5cc9a0ca6"
server_key_iv: "8617378c2c89d61478935247bb3f6d70"
server_key_version: "encryptionkey@0"
I20260620 14:52:13.640344 18429 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.003s sys 0.005s
I20260620 14:52:13.644435 18443 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 14:52:13.646077 18429 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260620 14:52:13.646319 18429 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskUci3DU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781966829091300-388-0/minicluster-data/ts-2/data,/tmp/dist-test-taskUci3DU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781966829091300-388-0/minicluster-data/ts-2/wal
uuid: "eae4e91c1b6e43aea4d1ba790802a0d9"
format_stamp: "Formatted at 2026-06-20 14:52:13 on dist-test-slave-q9j6"
server_key: "c31aacc2ff2c0dccfef02ea5cc9a0ca6"
server_key_iv: "8617378c2c89d61478935247bb3f6d70"
server_key_version: "encryptionkey@0"
I20260620 14:52:13.646584 18429 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskUci3DU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781966829091300-388-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskUci3DU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781966829091300-388-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskUci3DU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781966829091300-388-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 14:52:13.667877 18429 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 14:52:13.668871 18429 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 14:52:13.669199 18429 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 14:52:13.670382 18429 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 14:52:13.672063 18429 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 14:52:13.672201 18429 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 14:52:13.672350 18429 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 14:52:13.672442 18429 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 14:52:13.708348 18429 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.3:43197
I20260620 14:52:13.708413 18555 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.3:43197 every 8 connection(s)
I20260620 14:52:13.709749 18429 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskUci3DU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781966829091300-388-0/minicluster-data/ts-2/data/info.pb
I20260620 14:52:13.718061 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskUci3DU/build/asan/bin/kudu as pid 18429
I20260620 14:52:13.718246 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskUci3DU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781966829091300-388-0/minicluster-data/ts-2/wal/instance
I20260620 14:52:13.718768 388 external_mini_cluster.cc:1428] Setting key e93086e8d50627e6d4da048fe6b0268c
I20260620 14:52:13.723438 18297 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:52:13.712388 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.3:51645 (local address 127.0.97.62:45341)
0620 14:52:13.712564 (+ 176us) server_negotiation.cc:206] Beginning negotiation
0620 14:52:13.712569 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 14:52:13.713989 (+ 1420us) server_negotiation.cc:407] Connection header received
0620 14:52:13.715944 (+ 1955us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:52:13.715952 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:52:13.716059 (+ 107us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:52:13.716184 (+ 125us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:52:13.717336 (+ 1152us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:52:13.718425 (+ 1089us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:52:13.720369 (+ 1944us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:52:13.720920 (+ 551us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:52:13.721025 (+ 105us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:52:13.722726 (+ 1701us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 14:52:13.722731 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 14:52:13.722734 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 14:52:13.722763 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0620 14:52:13.722797 (+ 34us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 14:52:13.722810 (+ 13us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 14:52:13.722888 (+ 78us) server_negotiation.cc:1035] Waiting for connection context
0620 14:52:13.723098 (+ 210us) server_negotiation.cc:299] Negotiation successful
0620 14:52:13.723195 (+ 97us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":59}
I20260620 14:52:13.724606 18558 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:52:13.712862 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:45341 (local address 127.0.97.3:51645)
0620 14:52:13.713819 (+ 957us) negotiation.cc:107] Waiting for socket to connect
0620 14:52:13.713861 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0620 14:52:13.715566 (+ 1705us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 14:52:13.716435 (+ 869us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 14:52:13.716444 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 14:52:13.716547 (+ 103us) client_negotiation.cc:189] Negotiated authn=SASL
0620 14:52:13.717160 (+ 613us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:52:13.717178 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:52:13.718599 (+ 1421us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:52:13.718603 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:52:13.720208 (+ 1605us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:52:13.720232 (+ 24us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:52:13.721176 (+ 944us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:52:13.721186 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:52:13.721526 (+ 340us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:52:13.722377 (+ 851us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0620 14:52:13.722380 (+ 3us) client_negotiation.cc:656] calling sasl_client_start()
0620 14:52:13.722426 (+ 46us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0620 14:52:13.722521 (+ 95us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 14:52:13.722902 (+ 381us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 14:52:13.722910 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 14:52:13.722915 (+ 5us) client_negotiation.cc:769] Sending connection context
0620 14:52:13.723066 (+ 151us) client_negotiation.cc:240] Negotiation successful
0620 14:52:13.723428 (+ 362us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":482,"thread_start_us":286,"threads_started":1}
I20260620 14:52:13.726023 18556 heartbeater.cc:347] Connected to a master server at 127.0.97.62:45341
I20260620 14:52:13.726361 18556 heartbeater.cc:464] Registering TS with master...
I20260620 14:52:13.727152 18556 heartbeater.cc:511] Master 127.0.97.62:45341 requested a full tablet report, sending...
I20260620 14:52:13.728708 18109 ts_manager.cc:194] Registered new tserver with Master: eae4e91c1b6e43aea4d1ba790802a0d9 (127.0.97.3:43197)
I20260620 14:52:13.729512 18109 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.97.3:51645
I20260620 14:52:13.734407 388 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260620 14:52:13.748869 18297 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:52:13.742555 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39330 (local address 127.0.97.62:45341)
0620 14:52:13.742838 (+ 283us) server_negotiation.cc:206] Beginning negotiation
0620 14:52:13.742849 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0620 14:52:13.743211 (+ 362us) server_negotiation.cc:407] Connection header received
0620 14:52:13.743376 (+ 165us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:52:13.743381 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:52:13.743451 (+ 70us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:52:13.743591 (+ 140us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:52:13.744154 (+ 563us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:52:13.744894 (+ 740us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:52:13.746422 (+ 1528us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:52:13.746859 (+ 437us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:52:13.746965 (+ 106us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:52:13.748149 (+ 1184us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 14:52:13.748154 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 14:52:13.748157 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 14:52:13.748179 (+ 22us) server_negotiation.cc:922] calling sasl_server_start()
0620 14:52:13.748215 (+ 36us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 14:52:13.748227 (+ 12us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 14:52:13.748299 (+ 72us) server_negotiation.cc:1035] Waiting for connection context
0620 14:52:13.748422 (+ 123us) server_negotiation.cc:299] Negotiation successful
0620 14:52:13.748510 (+ 88us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":74}
I20260620 14:52:13.753129 18109 catalog_manager.cc:1908] Servicing CreateTable request from {username='slave'} at 127.0.0.1:39330:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260620 14:52:13.760488 18109 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260620 14:52:13.781489 18295 heartbeater.cc:503] Master 127.0.97.62:45341 was elected leader, sending a full tablet report...
I20260620 14:52:13.790352 18570 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:52:13.774215 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:38009 (local address 127.0.0.1:57038)
0620 14:52:13.776296 (+ 2081us) negotiation.cc:107] Waiting for socket to connect
0620 14:52:13.776353 (+ 57us) client_negotiation.cc:174] Beginning negotiation
0620 14:52:13.776588 (+ 235us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 14:52:13.780614 (+ 4026us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 14:52:13.780622 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 14:52:13.780658 (+ 36us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 14:52:13.780959 (+ 301us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:52:13.780973 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:52:13.784213 (+ 3240us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:52:13.784219 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:52:13.786366 (+ 2147us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:52:13.786383 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:52:13.789207 (+ 2824us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:52:13.789215 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:52:13.789481 (+ 266us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:52:13.789547 (+ 66us) client_negotiation.cc:769] Sending connection context
0620 14:52:13.789698 (+ 151us) client_negotiation.cc:240] Negotiation successful
0620 14:52:13.789950 (+ 252us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1700,"thread_start_us":2935,"threads_started":1}
I20260620 14:52:13.790463 18569 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:52:13.774499 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57038 (local address 127.0.97.1:38009)
0620 14:52:13.779312 (+ 4813us) server_negotiation.cc:206] Beginning negotiation
0620 14:52:13.779332 (+ 20us) server_negotiation.cc:399] Waiting for connection header
0620 14:52:13.779375 (+ 43us) server_negotiation.cc:407] Connection header received
0620 14:52:13.779502 (+ 127us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:52:13.779511 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:52:13.779915 (+ 404us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:52:13.781161 (+ 1246us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 14:52:13.782514 (+ 1353us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:52:13.784055 (+ 1541us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:52:13.786695 (+ 2640us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:52:13.788974 (+ 2279us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:52:13.789143 (+ 169us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:52:13.789493 (+ 350us) server_negotiation.cc:1035] Waiting for connection context
0620 14:52:13.789835 (+ 342us) server_negotiation.cc:299] Negotiation successful
0620 14:52:13.790060 (+ 225us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":4569,"thread_start_us":4424,"threads_started":1}
I20260620 14:52:13.790463 18568 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:52:13.774818 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.2:38417 (local address 127.0.0.1:38856)
0620 14:52:13.780271 (+ 5453us) negotiation.cc:107] Waiting for socket to connect
0620 14:52:13.780292 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0620 14:52:13.780415 (+ 123us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 14:52:13.783253 (+ 2838us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 14:52:13.783258 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 14:52:13.783282 (+ 24us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 14:52:13.783697 (+ 415us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:52:13.783715 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:52:13.785496 (+ 1781us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:52:13.785503 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:52:13.787531 (+ 2028us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:52:13.787546 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:52:13.789653 (+ 2107us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:52:13.789660 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:52:13.789855 (+ 195us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:52:13.789891 (+ 36us) client_negotiation.cc:769] Sending connection context
0620 14:52:13.789975 (+ 84us) client_negotiation.cc:240] Negotiation successful
0620 14:52:13.790134 (+ 159us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":2505,"thread_start_us":958,"threads_started":1}
I20260620 14:52:13.791160 18572 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:52:13.777889 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46380 (local address 127.0.97.3:43197)
0620 14:52:13.780790 (+ 2901us) server_negotiation.cc:206] Beginning negotiation
0620 14:52:13.780804 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0620 14:52:13.780849 (+ 45us) server_negotiation.cc:407] Connection header received
0620 14:52:13.780979 (+ 130us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:52:13.780991 (+ 12us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:52:13.781348 (+ 357us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:52:13.781632 (+ 284us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 14:52:13.783066 (+ 1434us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:52:13.784939 (+ 1873us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:52:13.788176 (+ 3237us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:52:13.789322 (+ 1146us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:52:13.789467 (+ 145us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:52:13.789754 (+ 287us) server_negotiation.cc:1035] Waiting for connection context
0620 14:52:13.790592 (+ 838us) server_negotiation.cc:299] Negotiation successful
0620 14:52:13.790800 (+ 208us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":2661,"thread_start_us":2471,"threads_started":1}
I20260620 14:52:13.791255 18571 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:52:13.775552 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38856 (local address 127.0.97.2:38417)
0620 14:52:13.782361 (+ 6809us) server_negotiation.cc:206] Beginning negotiation
0620 14:52:13.782369 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0620 14:52:13.782398 (+ 29us) server_negotiation.cc:407] Connection header received
0620 14:52:13.782493 (+ 95us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:52:13.782503 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:52:13.782778 (+ 275us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:52:13.782988 (+ 210us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 14:52:13.783918 (+ 930us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:52:13.785342 (+ 1424us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:52:13.787836 (+ 2494us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:52:13.789296 (+ 1460us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:52:13.789429 (+ 133us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:52:13.789694 (+ 265us) server_negotiation.cc:1035] Waiting for connection context
0620 14:52:13.790637 (+ 943us) server_negotiation.cc:299] Negotiation successful
0620 14:52:13.790872 (+ 235us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":6640,"thread_start_us":6504,"threads_started":1}
I20260620 14:52:13.791812 18573 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:52:13.778791 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.3:43197 (local address 127.0.0.1:46380)
0620 14:52:13.779829 (+ 1038us) negotiation.cc:107] Waiting for socket to connect
0620 14:52:13.779857 (+ 28us) client_negotiation.cc:174] Beginning negotiation
0620 14:52:13.780028 (+ 171us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 14:52:13.782383 (+ 2355us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 14:52:13.782388 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 14:52:13.782417 (+ 29us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 14:52:13.782843 (+ 426us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:52:13.782857 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:52:13.785100 (+ 2243us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:52:13.785106 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:52:13.787933 (+ 2827us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:52:13.787947 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:52:13.789896 (+ 1949us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:52:13.789902 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:52:13.790098 (+ 196us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:52:13.790136 (+ 38us) client_negotiation.cc:769] Sending connection context
0620 14:52:13.790221 (+ 85us) client_negotiation.cc:240] Negotiation successful
0620 14:52:13.790371 (+ 150us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":834,"thread_start_us":624,"threads_started":1}
I20260620 14:52:13.795583 18491 tablet_service.cc:1467] Processing CreateTablet for tablet 5a2b0555f66f4ff38dc295897823ef2f (DEFAULT_TABLE table=test-table [id=2f62adbbc81c46988563b34d2879925b]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 14:52:13.795511 18230 tablet_service.cc:1467] Processing CreateTablet for tablet 5a2b0555f66f4ff38dc295897823ef2f (DEFAULT_TABLE table=test-table [id=2f62adbbc81c46988563b34d2879925b]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 14:52:13.795511 18361 tablet_service.cc:1467] Processing CreateTablet for tablet 5a2b0555f66f4ff38dc295897823ef2f (DEFAULT_TABLE table=test-table [id=2f62adbbc81c46988563b34d2879925b]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 14:52:13.797071 18491 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5a2b0555f66f4ff38dc295897823ef2f. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 14:52:13.797118 18230 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5a2b0555f66f4ff38dc295897823ef2f. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 14:52:13.797119 18361 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5a2b0555f66f4ff38dc295897823ef2f. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 14:52:13.810878 18574 tablet_bootstrap.cc:492] T 5a2b0555f66f4ff38dc295897823ef2f P eae4e91c1b6e43aea4d1ba790802a0d9: Bootstrap starting.
I20260620 14:52:13.812975 18575 tablet_bootstrap.cc:492] T 5a2b0555f66f4ff38dc295897823ef2f P 6a51540ede984c80a500084f8c4e009f: Bootstrap starting.
I20260620 14:52:13.813892 18576 tablet_bootstrap.cc:492] T 5a2b0555f66f4ff38dc295897823ef2f P 8de773159a86446ea4716aa302c0e6ed: Bootstrap starting.
I20260620 14:52:13.816322 18574 tablet_bootstrap.cc:654] T 5a2b0555f66f4ff38dc295897823ef2f P eae4e91c1b6e43aea4d1ba790802a0d9: Neither blocks nor log segments found. Creating new log.
I20260620 14:52:13.817871 18574 log.cc:824] T 5a2b0555f66f4ff38dc295897823ef2f P eae4e91c1b6e43aea4d1ba790802a0d9: Log is configured to *not* fsync() on all Append() calls
I20260620 14:52:13.817966 18576 tablet_bootstrap.cc:654] T 5a2b0555f66f4ff38dc295897823ef2f P 8de773159a86446ea4716aa302c0e6ed: Neither blocks nor log segments found. Creating new log.
I20260620 14:52:13.819087 18575 tablet_bootstrap.cc:654] T 5a2b0555f66f4ff38dc295897823ef2f P 6a51540ede984c80a500084f8c4e009f: Neither blocks nor log segments found. Creating new log.
I20260620 14:52:13.819312 18576 log.cc:824] T 5a2b0555f66f4ff38dc295897823ef2f P 8de773159a86446ea4716aa302c0e6ed: Log is configured to *not* fsync() on all Append() calls
I20260620 14:52:13.820426 18575 log.cc:824] T 5a2b0555f66f4ff38dc295897823ef2f P 6a51540ede984c80a500084f8c4e009f: Log is configured to *not* fsync() on all Append() calls
I20260620 14:52:13.821585 18574 tablet_bootstrap.cc:492] T 5a2b0555f66f4ff38dc295897823ef2f P eae4e91c1b6e43aea4d1ba790802a0d9: No bootstrap required, opened a new log
I20260620 14:52:13.821931 18574 ts_tablet_manager.cc:1397] T 5a2b0555f66f4ff38dc295897823ef2f P eae4e91c1b6e43aea4d1ba790802a0d9: Time spent bootstrapping tablet: real 0.011s user 0.008s sys 0.000s
I20260620 14:52:13.822790 18576 tablet_bootstrap.cc:492] T 5a2b0555f66f4ff38dc295897823ef2f P 8de773159a86446ea4716aa302c0e6ed: No bootstrap required, opened a new log
I20260620 14:52:13.823099 18576 ts_tablet_manager.cc:1397] T 5a2b0555f66f4ff38dc295897823ef2f P 8de773159a86446ea4716aa302c0e6ed: Time spent bootstrapping tablet: real 0.009s user 0.006s sys 0.000s
I20260620 14:52:13.823328 18575 tablet_bootstrap.cc:492] T 5a2b0555f66f4ff38dc295897823ef2f P 6a51540ede984c80a500084f8c4e009f: No bootstrap required, opened a new log
I20260620 14:52:13.823649 18575 ts_tablet_manager.cc:1397] T 5a2b0555f66f4ff38dc295897823ef2f P 6a51540ede984c80a500084f8c4e009f: Time spent bootstrapping tablet: real 0.011s user 0.006s sys 0.000s
I20260620 14:52:13.831533 18575 raft_consensus.cc:348] T 5a2b0555f66f4ff38dc295897823ef2f P 6a51540ede984c80a500084f8c4e009f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6a51540ede984c80a500084f8c4e009f" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 38009 } } peers { permanent_uuid: "eae4e91c1b6e43aea4d1ba790802a0d9" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 43197 } } peers { permanent_uuid: "8de773159a86446ea4716aa302c0e6ed" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 38417 } }
I20260620 14:52:13.832093 18575 raft_consensus.cc:374] T 5a2b0555f66f4ff38dc295897823ef2f P 6a51540ede984c80a500084f8c4e009f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 14:52:13.832233 18575 raft_consensus.cc:729] T 5a2b0555f66f4ff38dc295897823ef2f P 6a51540ede984c80a500084f8c4e009f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6a51540ede984c80a500084f8c4e009f, State: Initialized, Role: FOLLOWER
I20260620 14:52:13.832931 18575 consensus_queue.cc:260] T 5a2b0555f66f4ff38dc295897823ef2f P 6a51540ede984c80a500084f8c4e009f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6a51540ede984c80a500084f8c4e009f" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 38009 } } peers { permanent_uuid: "eae4e91c1b6e43aea4d1ba790802a0d9" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 43197 } } peers { permanent_uuid: "8de773159a86446ea4716aa302c0e6ed" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 38417 } }
I20260620 14:52:13.833539 18574 raft_consensus.cc:348] T 5a2b0555f66f4ff38dc295897823ef2f P eae4e91c1b6e43aea4d1ba790802a0d9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6a51540ede984c80a500084f8c4e009f" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 38009 } } peers { permanent_uuid: "eae4e91c1b6e43aea4d1ba790802a0d9" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 43197 } } peers { permanent_uuid: "8de773159a86446ea4716aa302c0e6ed" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 38417 } }
I20260620 14:52:13.834273 18576 raft_consensus.cc:348] T 5a2b0555f66f4ff38dc295897823ef2f P 8de773159a86446ea4716aa302c0e6ed [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6a51540ede984c80a500084f8c4e009f" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 38009 } } peers { permanent_uuid: "eae4e91c1b6e43aea4d1ba790802a0d9" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 43197 } } peers { permanent_uuid: "8de773159a86446ea4716aa302c0e6ed" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 38417 } }
I20260620 14:52:13.834728 18576 raft_consensus.cc:374] T 5a2b0555f66f4ff38dc295897823ef2f P 8de773159a86446ea4716aa302c0e6ed [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 14:52:13.834828 18575 ts_tablet_manager.cc:1428] T 5a2b0555f66f4ff38dc295897823ef2f P 6a51540ede984c80a500084f8c4e009f: Time spent starting tablet: real 0.011s user 0.003s sys 0.007s
I20260620 14:52:13.834860 18574 raft_consensus.cc:374] T 5a2b0555f66f4ff38dc295897823ef2f P eae4e91c1b6e43aea4d1ba790802a0d9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 14:52:13.834914 18576 raft_consensus.cc:729] T 5a2b0555f66f4ff38dc295897823ef2f P 8de773159a86446ea4716aa302c0e6ed [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8de773159a86446ea4716aa302c0e6ed, State: Initialized, Role: FOLLOWER
I20260620 14:52:13.834993 18574 raft_consensus.cc:729] T 5a2b0555f66f4ff38dc295897823ef2f P eae4e91c1b6e43aea4d1ba790802a0d9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: eae4e91c1b6e43aea4d1ba790802a0d9, State: Initialized, Role: FOLLOWER
I20260620 14:52:13.835772 18574 consensus_queue.cc:260] T 5a2b0555f66f4ff38dc295897823ef2f P eae4e91c1b6e43aea4d1ba790802a0d9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6a51540ede984c80a500084f8c4e009f" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 38009 } } peers { permanent_uuid: "eae4e91c1b6e43aea4d1ba790802a0d9" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 43197 } } peers { permanent_uuid: "8de773159a86446ea4716aa302c0e6ed" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 38417 } }
I20260620 14:52:13.835813 18576 consensus_queue.cc:260] T 5a2b0555f66f4ff38dc295897823ef2f P 8de773159a86446ea4716aa302c0e6ed [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6a51540ede984c80a500084f8c4e009f" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 38009 } } peers { permanent_uuid: "eae4e91c1b6e43aea4d1ba790802a0d9" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 43197 } } peers { permanent_uuid: "8de773159a86446ea4716aa302c0e6ed" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 38417 } }
I20260620 14:52:13.836951 18556 heartbeater.cc:503] Master 127.0.97.62:45341 was elected leader, sending a full tablet report...
I20260620 14:52:13.837697 18574 ts_tablet_manager.cc:1428] T 5a2b0555f66f4ff38dc295897823ef2f P eae4e91c1b6e43aea4d1ba790802a0d9: Time spent starting tablet: real 0.016s user 0.012s sys 0.004s
I20260620 14:52:13.838802 18426 heartbeater.cc:503] Master 127.0.97.62:45341 was elected leader, sending a full tablet report...
I20260620 14:52:13.839489 18576 ts_tablet_manager.cc:1428] T 5a2b0555f66f4ff38dc295897823ef2f P 8de773159a86446ea4716aa302c0e6ed: Time spent starting tablet: real 0.016s user 0.017s sys 0.000s
W20260620 14:52:13.961592 18557 tablet.cc:2369] T 5a2b0555f66f4ff38dc295897823ef2f P eae4e91c1b6e43aea4d1ba790802a0d9: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260620 14:52:13.984367 18427 tablet.cc:2369] T 5a2b0555f66f4ff38dc295897823ef2f P 8de773159a86446ea4716aa302c0e6ed: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260620 14:52:14.007036 18296 tablet.cc:2369] T 5a2b0555f66f4ff38dc295897823ef2f P 6a51540ede984c80a500084f8c4e009f: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260620 14:52:14.167521 18580 raft_consensus.cc:482] T 5a2b0555f66f4ff38dc295897823ef2f P 6a51540ede984c80a500084f8c4e009f [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260620 14:52:14.167863 18580 raft_consensus.cc:504] T 5a2b0555f66f4ff38dc295897823ef2f P 6a51540ede984c80a500084f8c4e009f [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6a51540ede984c80a500084f8c4e009f" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 38009 } } peers { permanent_uuid: "eae4e91c1b6e43aea4d1ba790802a0d9" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 43197 } } peers { permanent_uuid: "8de773159a86446ea4716aa302c0e6ed" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 38417 } }
I20260620 14:52:14.169663 18580 leader_election.cc:288] T 5a2b0555f66f4ff38dc295897823ef2f P 6a51540ede984c80a500084f8c4e009f [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers eae4e91c1b6e43aea4d1ba790802a0d9 (127.0.97.3:43197), 8de773159a86446ea4716aa302c0e6ed (127.0.97.2:38417)
I20260620 14:52:14.177024 18571 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:52:14.170159 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.1:49935 (local address 127.0.97.2:38417)
0620 14:52:14.170548 (+ 389us) server_negotiation.cc:206] Beginning negotiation
0620 14:52:14.170554 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 14:52:14.171173 (+ 619us) server_negotiation.cc:407] Connection header received
0620 14:52:14.171357 (+ 184us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:52:14.171361 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:52:14.171481 (+ 120us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:52:14.171584 (+ 103us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 14:52:14.172257 (+ 673us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:52:14.172975 (+ 718us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:52:14.175109 (+ 2134us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:52:14.175974 (+ 865us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:52:14.176107 (+ 133us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:52:14.176217 (+ 110us) server_negotiation.cc:1035] Waiting for connection context
0620 14:52:14.176587 (+ 370us) server_negotiation.cc:299] Negotiation successful
0620 14:52:14.176730 (+ 143us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":268}
I20260620 14:52:14.177019 18572 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:52:14.170102 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.1:58185 (local address 127.0.97.3:43197)
0620 14:52:14.170379 (+ 277us) server_negotiation.cc:206] Beginning negotiation
0620 14:52:14.170388 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0620 14:52:14.170997 (+ 609us) server_negotiation.cc:407] Connection header received
0620 14:52:14.171218 (+ 221us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:52:14.171224 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:52:14.171309 (+ 85us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:52:14.171397 (+ 88us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 14:52:14.172117 (+ 720us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:52:14.172889 (+ 772us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:52:14.175144 (+ 2255us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:52:14.176002 (+ 858us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:52:14.176166 (+ 164us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:52:14.176263 (+ 97us) server_negotiation.cc:1035] Waiting for connection context
0620 14:52:14.176648 (+ 385us) server_negotiation.cc:299] Negotiation successful
0620 14:52:14.176787 (+ 139us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":95}
I20260620 14:52:14.177172 18584 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:52:14.170128 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.2:38417 (local address 127.0.97.1:49935)
0620 14:52:14.171031 (+ 903us) negotiation.cc:107] Waiting for socket to connect
0620 14:52:14.171052 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0620 14:52:14.171212 (+ 160us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 14:52:14.171674 (+ 462us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 14:52:14.171678 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 14:52:14.171699 (+ 21us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 14:52:14.172075 (+ 376us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:52:14.172089 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:52:14.173121 (+ 1032us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:52:14.173125 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:52:14.174864 (+ 1739us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:52:14.174879 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:52:14.176188 (+ 1309us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:52:14.176195 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:52:14.176391 (+ 196us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:52:14.176439 (+ 48us) client_negotiation.cc:769] Sending connection context
0620 14:52:14.176518 (+ 79us) client_negotiation.cc:240] Negotiation successful
0620 14:52:14.176685 (+ 167us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":749,"mutex_wait_us":109,"thread_start_us":571,"threads_started":1}
I20260620 14:52:14.177202 18583 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:52:14.169894 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.3:43197 (local address 127.0.97.1:58185)
0620 14:52:14.170836 (+ 942us) negotiation.cc:107] Waiting for socket to connect
0620 14:52:14.170861 (+ 25us) client_negotiation.cc:174] Beginning negotiation
0620 14:52:14.171051 (+ 190us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 14:52:14.171446 (+ 395us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 14:52:14.171451 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 14:52:14.171475 (+ 24us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 14:52:14.171943 (+ 468us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:52:14.171958 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:52:14.173034 (+ 1076us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:52:14.173042 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:52:14.174879 (+ 1837us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:52:14.174893 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:52:14.176239 (+ 1346us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:52:14.176248 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:52:14.176404 (+ 156us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:52:14.176476 (+ 72us) client_negotiation.cc:769] Sending connection context
0620 14:52:14.176598 (+ 122us) client_negotiation.cc:240] Negotiation successful
0620 14:52:14.176731 (+ 133us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":757,"mutex_wait_us":166,"thread_start_us":518,"threads_started":1}
I20260620 14:52:14.178500 18511 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5a2b0555f66f4ff38dc295897823ef2f" candidate_uuid: "6a51540ede984c80a500084f8c4e009f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "eae4e91c1b6e43aea4d1ba790802a0d9" is_pre_election: true
I20260620 14:52:14.178561 18381 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5a2b0555f66f4ff38dc295897823ef2f" candidate_uuid: "6a51540ede984c80a500084f8c4e009f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8de773159a86446ea4716aa302c0e6ed" is_pre_election: true
I20260620 14:52:14.179065 18511 raft_consensus.cc:2454] T 5a2b0555f66f4ff38dc295897823ef2f P eae4e91c1b6e43aea4d1ba790802a0d9 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6a51540ede984c80a500084f8c4e009f in term 0.
I20260620 14:52:14.179087 18381 raft_consensus.cc:2454] T 5a2b0555f66f4ff38dc295897823ef2f P 8de773159a86446ea4716aa302c0e6ed [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6a51540ede984c80a500084f8c4e009f in term 0.
I20260620 14:52:14.179865 18183 leader_election.cc:302] T 5a2b0555f66f4ff38dc295897823ef2f P 6a51540ede984c80a500084f8c4e009f [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6a51540ede984c80a500084f8c4e009f, 8de773159a86446ea4716aa302c0e6ed; no voters:
I20260620 14:52:14.180395 18580 raft_consensus.cc:2781] T 5a2b0555f66f4ff38dc295897823ef2f P 6a51540ede984c80a500084f8c4e009f [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260620 14:52:14.180573 18580 raft_consensus.cc:482] T 5a2b0555f66f4ff38dc295897823ef2f P 6a51540ede984c80a500084f8c4e009f [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260620 14:52:14.180713 18580 raft_consensus.cc:3037] T 5a2b0555f66f4ff38dc295897823ef2f P 6a51540ede984c80a500084f8c4e009f [term 0 FOLLOWER]: Advancing to term 1
I20260620 14:52:14.183022 18580 raft_consensus.cc:504] T 5a2b0555f66f4ff38dc295897823ef2f P 6a51540ede984c80a500084f8c4e009f [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6a51540ede984c80a500084f8c4e009f" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 38009 } } peers { permanent_uuid: "eae4e91c1b6e43aea4d1ba790802a0d9" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 43197 } } peers { permanent_uuid: "8de773159a86446ea4716aa302c0e6ed" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 38417 } }
I20260620 14:52:14.183841 18580 leader_election.cc:288] T 5a2b0555f66f4ff38dc295897823ef2f P 6a51540ede984c80a500084f8c4e009f [CANDIDATE]: Term 1 election: Requested vote from peers eae4e91c1b6e43aea4d1ba790802a0d9 (127.0.97.3:43197), 8de773159a86446ea4716aa302c0e6ed (127.0.97.2:38417)
I20260620 14:52:14.184367 18511 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5a2b0555f66f4ff38dc295897823ef2f" candidate_uuid: "6a51540ede984c80a500084f8c4e009f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "eae4e91c1b6e43aea4d1ba790802a0d9"
I20260620 14:52:14.184453 18381 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5a2b0555f66f4ff38dc295897823ef2f" candidate_uuid: "6a51540ede984c80a500084f8c4e009f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8de773159a86446ea4716aa302c0e6ed"
I20260620 14:52:14.184664 18511 raft_consensus.cc:3037] T 5a2b0555f66f4ff38dc295897823ef2f P eae4e91c1b6e43aea4d1ba790802a0d9 [term 0 FOLLOWER]: Advancing to term 1
I20260620 14:52:14.184664 18381 raft_consensus.cc:3037] T 5a2b0555f66f4ff38dc295897823ef2f P 8de773159a86446ea4716aa302c0e6ed [term 0 FOLLOWER]: Advancing to term 1
I20260620 14:52:14.188283 18381 raft_consensus.cc:2454] T 5a2b0555f66f4ff38dc295897823ef2f P 8de773159a86446ea4716aa302c0e6ed [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6a51540ede984c80a500084f8c4e009f in term 1.
I20260620 14:52:14.188289 18511 raft_consensus.cc:2454] T 5a2b0555f66f4ff38dc295897823ef2f P eae4e91c1b6e43aea4d1ba790802a0d9 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6a51540ede984c80a500084f8c4e009f in term 1.
I20260620 14:52:14.188908 18183 leader_election.cc:302] T 5a2b0555f66f4ff38dc295897823ef2f P 6a51540ede984c80a500084f8c4e009f [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 3 yes votes; 0 no votes. yes voters: 6a51540ede984c80a500084f8c4e009f, 8de773159a86446ea4716aa302c0e6ed, eae4e91c1b6e43aea4d1ba790802a0d9; no voters:
I20260620 14:52:14.189348 18580 raft_consensus.cc:2781] T 5a2b0555f66f4ff38dc295897823ef2f P 6a51540ede984c80a500084f8c4e009f [term 1 FOLLOWER]: Leader election won for term 1
I20260620 14:52:14.189857 18580 raft_consensus.cc:686] T 5a2b0555f66f4ff38dc295897823ef2f P 6a51540ede984c80a500084f8c4e009f [term 1 LEADER]: Becoming Leader. State: Replica: 6a51540ede984c80a500084f8c4e009f, State: Running, Role: LEADER
I20260620 14:52:14.190431 18580 consensus_queue.cc:237] T 5a2b0555f66f4ff38dc295897823ef2f P 6a51540ede984c80a500084f8c4e009f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6a51540ede984c80a500084f8c4e009f" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 38009 } } peers { permanent_uuid: "eae4e91c1b6e43aea4d1ba790802a0d9" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 43197 } } peers { permanent_uuid: "8de773159a86446ea4716aa302c0e6ed" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 38417 } }
I20260620 14:52:14.195394 18107 catalog_manager.cc:5495] T 5a2b0555f66f4ff38dc295897823ef2f P 6a51540ede984c80a500084f8c4e009f reported cstate change: term changed from 0 to 1, leader changed from <none> to 6a51540ede984c80a500084f8c4e009f (127.0.97.1). New cstate: current_term: 1 leader_uuid: "6a51540ede984c80a500084f8c4e009f" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6a51540ede984c80a500084f8c4e009f" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 38009 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "eae4e91c1b6e43aea4d1ba790802a0d9" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 43197 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "8de773159a86446ea4716aa302c0e6ed" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 38417 } health_report { overall_health: UNKNOWN } } }
I20260620 14:52:14.246212 18569 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:52:14.240509 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57040 (local address 127.0.97.1:38009)
0620 14:52:14.240708 (+ 199us) server_negotiation.cc:206] Beginning negotiation
0620 14:52:14.240714 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 14:52:14.240735 (+ 21us) server_negotiation.cc:407] Connection header received
0620 14:52:14.240926 (+ 191us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:52:14.240932 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:52:14.241139 (+ 207us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:52:14.241314 (+ 175us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 14:52:14.241707 (+ 393us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:52:14.242429 (+ 722us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:52:14.243947 (+ 1518us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:52:14.244398 (+ 451us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:52:14.244544 (+ 146us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:52:14.245075 (+ 531us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 14:52:14.245606 (+ 531us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0620 14:52:14.245775 (+ 169us) server_negotiation.cc:1035] Waiting for connection context
0620 14:52:14.245856 (+ 81us) server_negotiation.cc:299] Negotiation successful
0620 14:52:14.245947 (+ 91us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":48}
I20260620 14:52:14.255477 18381 raft_consensus.cc:1261] T 5a2b0555f66f4ff38dc295897823ef2f P 8de773159a86446ea4716aa302c0e6ed [term 1 FOLLOWER]: Refusing update from remote peer 6a51540ede984c80a500084f8c4e009f: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 14:52:14.255477 18511 raft_consensus.cc:1261] T 5a2b0555f66f4ff38dc295897823ef2f P eae4e91c1b6e43aea4d1ba790802a0d9 [term 1 FOLLOWER]: Refusing update from remote peer 6a51540ede984c80a500084f8c4e009f: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 14:52:14.256771 18585 consensus_queue.cc:1035] T 5a2b0555f66f4ff38dc295897823ef2f P 6a51540ede984c80a500084f8c4e009f [LEADER]: Connected to new peer: Peer: permanent_uuid: "8de773159a86446ea4716aa302c0e6ed" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 38417 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 14:52:14.257150 18580 consensus_queue.cc:1035] T 5a2b0555f66f4ff38dc295897823ef2f P 6a51540ede984c80a500084f8c4e009f [LEADER]: Connected to new peer: Peer: permanent_uuid: "eae4e91c1b6e43aea4d1ba790802a0d9" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 43197 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 14:52:14.270678 18588 mvcc.cc:205] Tried to move back new op lower bound from 7298937381898866688 to 7298937381648207872. Current Snapshot: MvccSnapshot[applied={T|T < 7298937381898866688}]
I20260620 14:52:14.273785 18589 mvcc.cc:205] Tried to move back new op lower bound from 7298937381898866688 to 7298937381648207872. Current Snapshot: MvccSnapshot[applied={T|T < 7298937381898866688}]
I20260620 14:52:14.275148 18590 mvcc.cc:205] Tried to move back new op lower bound from 7298937381898866688 to 7298937381648207872. Current Snapshot: MvccSnapshot[applied={T|T < 7298937381898866688}]
I20260620 14:52:14.286593 18108 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:39330:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 14:52:14.286888 18108 catalog_manager.cc:2645] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:39330:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 14:52:14.290855 18108 catalog_manager.cc:5782] T 00000000000000000000000000000000 P acb0b0114bf342d59b50f175139edc47: Sending DeleteTablet for 3 replicas of tablet 5a2b0555f66f4ff38dc295897823ef2f
I20260620 14:52:14.292270 18230 tablet_service.cc:1514] Processing DeleteTablet for tablet 5a2b0555f66f4ff38dc295897823ef2f with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 14:52:14 UTC) from {username='slave'} at 127.0.0.1:57038
I20260620 14:52:14.292312 18361 tablet_service.cc:1514] Processing DeleteTablet for tablet 5a2b0555f66f4ff38dc295897823ef2f with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 14:52:14 UTC) from {username='slave'} at 127.0.0.1:38856
I20260620 14:52:14.293502 18599 tablet_replica.cc:330] T 5a2b0555f66f4ff38dc295897823ef2f P 8de773159a86446ea4716aa302c0e6ed: stopping tablet replica
I20260620 14:52:14.293454 18491 tablet_service.cc:1514] Processing DeleteTablet for tablet 5a2b0555f66f4ff38dc295897823ef2f with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 14:52:14 UTC) from {username='slave'} at 127.0.0.1:46380
I20260620 14:52:14.294060 18599 raft_consensus.cc:2229] T 5a2b0555f66f4ff38dc295897823ef2f P 8de773159a86446ea4716aa302c0e6ed [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 14:52:14.294421 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskUci3DU/build/asan/bin/kudu pid 18168
I20260620 14:52:14.294787 18599 raft_consensus.cc:2258] T 5a2b0555f66f4ff38dc295897823ef2f P 8de773159a86446ea4716aa302c0e6ed [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 14:52:14.302603 18599 ts_tablet_manager.cc:1905] T 5a2b0555f66f4ff38dc295897823ef2f P 8de773159a86446ea4716aa302c0e6ed: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 14:52:14.306178 18600 tablet_replica.cc:330] T 5a2b0555f66f4ff38dc295897823ef2f P eae4e91c1b6e43aea4d1ba790802a0d9: stopping tablet replica
I20260620 14:52:14.306696 18600 raft_consensus.cc:2229] T 5a2b0555f66f4ff38dc295897823ef2f P eae4e91c1b6e43aea4d1ba790802a0d9 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 14:52:14.307268 18600 raft_consensus.cc:2258] T 5a2b0555f66f4ff38dc295897823ef2f P eae4e91c1b6e43aea4d1ba790802a0d9 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 14:52:14.309801 18598 tablet_replica.cc:330] T 5a2b0555f66f4ff38dc295897823ef2f P 6a51540ede984c80a500084f8c4e009f: stopping tablet replica
I20260620 14:52:14.310446 18598 raft_consensus.cc:2229] T 5a2b0555f66f4ff38dc295897823ef2f P 6a51540ede984c80a500084f8c4e009f [term 1 LEADER]: Raft consensus shutting down.
I20260620 14:52:14.311291 18598 raft_consensus.cc:2258] T 5a2b0555f66f4ff38dc295897823ef2f P 6a51540ede984c80a500084f8c4e009f [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 14:52:14.313764 18600 ts_tablet_manager.cc:1905] T 5a2b0555f66f4ff38dc295897823ef2f P eae4e91c1b6e43aea4d1ba790802a0d9: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 14:52:14.314131 18599 ts_tablet_manager.cc:1918] T 5a2b0555f66f4ff38dc295897823ef2f P 8de773159a86446ea4716aa302c0e6ed: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260620 14:52:14.314452 18599 log.cc:1196] T 5a2b0555f66f4ff38dc295897823ef2f P 8de773159a86446ea4716aa302c0e6ed: Deleting WAL directory at /tmp/dist-test-taskUci3DU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781966829091300-388-0/minicluster-data/ts-1/wal/wals/5a2b0555f66f4ff38dc295897823ef2f
I20260620 14:52:14.316066 18569 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:52:14.296891 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57050 (local address 127.0.97.1:38009)
0620 14:52:14.297109 (+ 218us) server_negotiation.cc:206] Beginning negotiation
0620 14:52:14.297116 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0620 14:52:14.306230 (+ 9114us) server_negotiation.cc:407] Connection header received
0620 14:52:14.306799 (+ 569us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:52:14.306807 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:52:14.306954 (+ 147us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:52:14.307039 (+ 85us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:52:14.307982 (+ 943us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:52:14.309274 (+ 1292us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:52:14.311398 (+ 2124us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:52:14.312198 (+ 800us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:52:14.312323 (+ 125us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:52:14.314006 (+ 1683us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 14:52:14.314020 (+ 14us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 14:52:14.314040 (+ 20us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 14:52:14.315007 (+ 967us) server_negotiation.cc:922] calling sasl_server_start()
0620 14:52:14.315073 (+ 66us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 14:52:14.315116 (+ 43us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 14:52:14.315207 (+ 91us) server_negotiation.cc:1035] Waiting for connection context
0620 14:52:14.315521 (+ 314us) server_negotiation.cc:299] Negotiation successful
0620 14:52:14.315671 (+ 150us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":67}
I20260620 14:52:14.316092 18598 ts_tablet_manager.cc:1905] T 5a2b0555f66f4ff38dc295897823ef2f P 6a51540ede984c80a500084f8c4e009f: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 14:52:14.316787 18599 ts_tablet_manager.cc:1939] T 5a2b0555f66f4ff38dc295897823ef2f P 8de773159a86446ea4716aa302c0e6ed: Deleting consensus metadata
I20260620 14:52:14.318390 18290 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 14:52:14.318919 18093 catalog_manager.cc:4841] TS 8de773159a86446ea4716aa302c0e6ed (127.0.97.2:38417): tablet 5a2b0555f66f4ff38dc295897823ef2f (table test-table [id=2f62adbbc81c46988563b34d2879925b]) successfully deleted
I20260620 14:52:14.322860 18600 ts_tablet_manager.cc:1918] T 5a2b0555f66f4ff38dc295897823ef2f P eae4e91c1b6e43aea4d1ba790802a0d9: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260620 14:52:14.323136 18600 log.cc:1196] T 5a2b0555f66f4ff38dc295897823ef2f P eae4e91c1b6e43aea4d1ba790802a0d9: Deleting WAL directory at /tmp/dist-test-taskUci3DU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781966829091300-388-0/minicluster-data/ts-2/wal/wals/5a2b0555f66f4ff38dc295897823ef2f
I20260620 14:52:14.323750 18600 ts_tablet_manager.cc:1939] T 5a2b0555f66f4ff38dc295897823ef2f P eae4e91c1b6e43aea4d1ba790802a0d9: Deleting consensus metadata
I20260620 14:52:14.325815 18096 catalog_manager.cc:4841] TS eae4e91c1b6e43aea4d1ba790802a0d9 (127.0.97.3:43197): tablet 5a2b0555f66f4ff38dc295897823ef2f (table test-table [id=2f62adbbc81c46988563b34d2879925b]) successfully deleted
I20260620 14:52:14.421475 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskUci3DU/build/asan/bin/kudu with pid 18168
W20260620 14:52:14.440341 18096 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.0.97.1:38009 (error 108)
I20260620 14:52:14.440516 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskUci3DU/build/asan/bin/kudu pid 18299
I20260620 14:52:14.442742 18604 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:52:14.441903 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:38009 (local address 127.0.0.1:57056)
0620 14:52:14.442418 (+ 515us) negotiation.cc:107] Waiting for socket to connect
0620 14:52:14.442549 (+ 131us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.1:38009: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":378,"thread_start_us":285,"threads_started":1}
W20260620 14:52:14.443027 18096 catalog_manager.cc:4584] TS 6a51540ede984c80a500084f8c4e009f (127.0.97.1:38009): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 5a2b0555f66f4ff38dc295897823ef2f: Network error: Client connection negotiation failed: client connection to 127.0.97.1:38009: connect: Connection refused (error 111)
I20260620 14:52:14.448206 18571 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:52:14.441281 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38866 (local address 127.0.97.2:38417)
0620 14:52:14.441456 (+ 175us) server_negotiation.cc:206] Beginning negotiation
0620 14:52:14.441461 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 14:52:14.441559 (+ 98us) server_negotiation.cc:407] Connection header received
0620 14:52:14.441737 (+ 178us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:52:14.441742 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:52:14.441830 (+ 88us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:52:14.441953 (+ 123us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:52:14.442418 (+ 465us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:52:14.443314 (+ 896us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:52:14.444749 (+ 1435us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:52:14.445164 (+ 415us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:52:14.445318 (+ 154us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:52:14.446517 (+ 1199us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 14:52:14.446528 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 14:52:14.446544 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 14:52:14.447178 (+ 634us) server_negotiation.cc:922] calling sasl_server_start()
0620 14:52:14.447248 (+ 70us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 14:52:14.447270 (+ 22us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 14:52:14.447402 (+ 132us) server_negotiation.cc:1035] Waiting for connection context
0620 14:52:14.447582 (+ 180us) server_negotiation.cc:299] Negotiation successful
0620 14:52:14.447902 (+ 320us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":65}
I20260620 14:52:14.449018 18421 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 14:52:14.496718 18604 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:52:14.496136 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:38009 (local address 127.0.0.1:57070)
0620 14:52:14.496416 (+ 280us) negotiation.cc:107] Waiting for socket to connect
0620 14:52:14.496531 (+ 115us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.1:38009: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":119}
I20260620 14:52:14.497891 18604 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:52:14.497454 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:38009 (local address 127.0.0.1:57076)
0620 14:52:14.497621 (+ 167us) negotiation.cc:107] Waiting for socket to connect
0620 14:52:14.497712 (+ 91us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.1:38009: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":55}
I20260620 14:52:14.540654 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskUci3DU/build/asan/bin/kudu with pid 18299
I20260620 14:52:14.558467 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskUci3DU/build/asan/bin/kudu pid 18429
I20260620 14:52:14.558529 18604 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:52:14.558033 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:38009 (local address 127.0.0.1:57084)
0620 14:52:14.558289 (+ 256us) negotiation.cc:107] Waiting for socket to connect
0620 14:52:14.558381 (+ 92us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.1:38009: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":80}
I20260620 14:52:14.559782 18604 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:52:14.559235 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:38009 (local address 127.0.0.1:57090)
0620 14:52:14.559483 (+ 248us) negotiation.cc:107] Waiting for socket to connect
0620 14:52:14.559566 (+ 83us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.1:38009: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":143}
I20260620 14:52:14.565538 18572 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:52:14.559302 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46388 (local address 127.0.97.3:43197)
0620 14:52:14.559488 (+ 186us) server_negotiation.cc:206] Beginning negotiation
0620 14:52:14.559496 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0620 14:52:14.559518 (+ 22us) server_negotiation.cc:407] Connection header received
0620 14:52:14.559718 (+ 200us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:52:14.559724 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:52:14.559855 (+ 131us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:52:14.559968 (+ 113us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:52:14.560416 (+ 448us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:52:14.561139 (+ 723us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:52:14.562494 (+ 1355us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:52:14.562892 (+ 398us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:52:14.563030 (+ 138us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:52:14.564190 (+ 1160us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 14:52:14.564199 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 14:52:14.564214 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 14:52:14.564769 (+ 555us) server_negotiation.cc:922] calling sasl_server_start()
0620 14:52:14.564817 (+ 48us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 14:52:14.564835 (+ 18us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 14:52:14.564967 (+ 132us) server_negotiation.cc:1035] Waiting for connection context
0620 14:52:14.565104 (+ 137us) server_negotiation.cc:299] Negotiation successful
0620 14:52:14.565218 (+ 114us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":59}
I20260620 14:52:14.566179 18551 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 14:52:14.640754 18604 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:52:14.640222 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:38009 (local address 127.0.0.1:57092)
0620 14:52:14.640480 (+ 258us) negotiation.cc:107] Waiting for socket to connect
0620 14:52:14.640576 (+ 96us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.1:38009: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":94}
I20260620 14:52:14.642009 18604 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:52:14.641563 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:38009 (local address 127.0.0.1:57098)
0620 14:52:14.641787 (+ 224us) negotiation.cc:107] Waiting for socket to connect
0620 14:52:14.641865 (+ 78us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.1:38009: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":125}
I20260620 14:52:14.661047 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskUci3DU/build/asan/bin/kudu with pid 18429
I20260620 14:52:14.678129 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskUci3DU/build/asan/bin/kudu pid 18078
I20260620 14:52:14.679298 18139 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 14:52:14.791131 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskUci3DU/build/asan/bin/kudu with pid 18078
2026-06-20T14:52:14Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/1 (3184 ms)
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/2
Loading random data
Initializing database '/tmp/dist-test-taskUci3DU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781966829091300-388-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 20 14:52:14 dist-test-slave-q9j6 krb5kdc[18610](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 20 14:52:14 dist-test-slave-q9j6 krb5kdc[18610](info): set up 2 sockets
Jun 20 14:52:14 dist-test-slave-q9j6 krb5kdc[18610](info): commencing operation
krb5kdc: starting...
W20260620 14:52:15.907886 388 mini_kdc.cc:121] Time spent starting KDC: real 1.074s user 0.000s sys 0.027s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskUci3DU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781966829091300-388-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskUci3DU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781966829091300-388-0/krb5kdc/test-user.keytab.
Jun 20 14:52:16 dist-test-slave-q9j6 krb5kdc[18610](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781967136, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-20T14:52:16Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T14:52:16Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.97.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskUci3DU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781966829091300-388-0/krb5kdc/kudu_127.0.97.62.keytab.
Entry for principal kudu/127.0.97.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskUci3DU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781966829091300-388-0/krb5kdc/kudu_127.0.97.62.keytab.
I20260620 14:52:16.168654 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskUci3DU/build/asan/bin/kudu
/tmp/dist-test-taskUci3DU/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskUci3DU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781966829091300-388-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskUci3DU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781966829091300-388-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskUci3DU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781966829091300-388-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskUci3DU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781966829091300-388-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:38439
--webserver_interface=127.0.97.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskUci3DU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781966829091300-388-0/krb5kdc/kudu_127.0.97.62.keytab
--principal=kudu/127.0.97.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:42235
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.97.62:38439
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskUci3DU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781966829091300-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskUci3DU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781966829091300-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskUci3DU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781966829091300-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 14:52:16.513170 18623 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 14:52:16.513495 18623 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 14:52:16.513697 18623 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 14:52:16.521186 18623 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 14:52:16.521314 18623 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 14:52:16.521413 18623 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 14:52:16.521533 18623 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 14:52:16.521592 18623 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 14:52:16.531903 18623 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:42235
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskUci3DU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781966829091300-388-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskUci3DU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781966829091300-388-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.97.62:38439
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskUci3DU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781966829091300-388-0/krb5kdc/kudu_127.0.97.62.keytab
--principal=kudu/127.0.97.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:38439
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskUci3DU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781966829091300-388-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.97.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskUci3DU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781966829091300-388-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 13:41:20 UTC on ccycloud.quasar-vdagoi.root.comops.site
build id 4457462
ASAN enabled
I20260620 14:52:16.533394 18623 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 14:52:16.534838 18623 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 14:52:16.547454 18629 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 14:52:16.547550 18631 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 14:52:16.547470 18628 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 14:52:16.548780 18623 server_base.cc:1034] running on GCE node
I20260620 14:52:16.549705 18623 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 14:52:16.551292 18623 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 14:52:16.552556 18623 hybrid_clock.cc:648] HybridClock initialized: now 1781967136552474 us; error 70 us; skew 500 ppm
I20260620 14:52:16.552973 18623 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 14:52:16 dist-test-slave-q9j6 krb5kdc[18610](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781967136, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 14:52:16.556406 18623 init.cc:377] Logged in from keytab as kudu/127.0.97.62@KRBTEST.COM (short username kudu)
I20260620 14:52:16.558049 18623 webserver.cc:460] Webserver started at http://127.0.97.62:42563/ using document root <none> and password file <none>
I20260620 14:52:16.558812 18623 fs_manager.cc:359] Metadata directory not provided
I20260620 14:52:16.558960 18623 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 14:52:16.559295 18623 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 14:52:16.562578 18623 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskUci3DU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781966829091300-388-0/minicluster-data/master-0/data/instance:
uuid: "92fc2fdb55334e25980592943ab9f8f2"
format_stamp: "Formatted at 2026-06-20 14:52:16 on dist-test-slave-q9j6"
server_key: "87be859396e66358c3cbca7354d3c25b"
server_key_iv: "8f81d014c599154b575c42cff2a92df2"
server_key_version: "encryptionkey@0"
I20260620 14:52:16.563432 18623 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskUci3DU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781966829091300-388-0/minicluster-data/master-0/wal/instance:
uuid: "92fc2fdb55334e25980592943ab9f8f2"
format_stamp: "Formatted at 2026-06-20 14:52:16 on dist-test-slave-q9j6"
server_key: "87be859396e66358c3cbca7354d3c25b"
server_key_iv: "8f81d014c599154b575c42cff2a92df2"
server_key_version: "encryptionkey@0"
I20260620 14:52:16.569207 18623 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.004s sys 0.000s
I20260620 14:52:16.573395 18638 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 14:52:16.575119 18623 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.004s sys 0.000s
I20260620 14:52:16.575381 18623 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskUci3DU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781966829091300-388-0/minicluster-data/master-0/data,/tmp/dist-test-taskUci3DU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781966829091300-388-0/minicluster-data/master-0/wal
uuid: "92fc2fdb55334e25980592943ab9f8f2"
format_stamp: "Formatted at 2026-06-20 14:52:16 on dist-test-slave-q9j6"
server_key: "87be859396e66358c3cbca7354d3c25b"
server_key_iv: "8f81d014c599154b575c42cff2a92df2"
server_key_version: "encryptionkey@0"
I20260620 14:52:16.575680 18623 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskUci3DU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781966829091300-388-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskUci3DU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781966829091300-388-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskUci3DU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781966829091300-388-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 14:52:16.610033 18623 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 14:52:16.616266 18623 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 14:52:16.616652 18623 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 14:52:16.638831 18623 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.62:38439
I20260620 14:52:16.638854 18689 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.62:38439 every 8 connection(s)
I20260620 14:52:16.640448 18623 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskUci3DU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781966829091300-388-0/minicluster-data/master-0/data/info.pb
I20260620 14:52:16.644546 18690 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 14:52:16.644814 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskUci3DU/build/asan/bin/kudu as pid 18623
I20260620 14:52:16.645010 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskUci3DU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781966829091300-388-0/minicluster-data/master-0/wal/instance
I20260620 14:52:16.645457 388 external_mini_cluster.cc:1428] Setting key ad94afb9bccc4972e9e1e0597ef9e871
I20260620 14:52:16.660003 18690 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 92fc2fdb55334e25980592943ab9f8f2: Bootstrap starting.
Jun 20 14:52:16 dist-test-slave-q9j6 krb5kdc[18610](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781967136, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260620 14:52:16.664173 18690 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 92fc2fdb55334e25980592943ab9f8f2: Neither blocks nor log segments found. Creating new log.
I20260620 14:52:16.665453 18690 log.cc:824] T 00000000000000000000000000000000 P 92fc2fdb55334e25980592943ab9f8f2: Log is configured to *not* fsync() on all Append() calls
I20260620 14:52:16.668851 18690 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 92fc2fdb55334e25980592943ab9f8f2: No bootstrap required, opened a new log
I20260620 14:52:16.672014 18693 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:52:16.647407 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44514 (local address 127.0.97.62:38439)
0620 14:52:16.648367 (+ 960us) server_negotiation.cc:206] Beginning negotiation
0620 14:52:16.648379 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0620 14:52:16.648424 (+ 45us) server_negotiation.cc:407] Connection header received
0620 14:52:16.650852 (+ 2428us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:52:16.650883 (+ 31us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:52:16.651352 (+ 469us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:52:16.651825 (+ 473us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:52:16.653611 (+ 1786us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:52:16.655037 (+ 1426us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:52:16.657210 (+ 2173us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:52:16.658008 (+ 798us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:52:16.658164 (+ 156us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:52:16.663469 (+ 5305us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 14:52:16.663479 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 14:52:16.663498 (+ 19us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 14:52:16.663554 (+ 56us) server_negotiation.cc:922] calling sasl_server_start()
0620 14:52:16.667123 (+ 3569us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:52:16.668111 (+ 988us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:52:16.668119 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:52:16.668127 (+ 8us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:52:16.668329 (+ 202us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:52:16.668789 (+ 460us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:52:16.668793 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:52:16.668795 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:52:16.669448 (+ 653us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 14:52:16.669784 (+ 336us) server_negotiation.cc:1035] Waiting for connection context
0620 14:52:16.670276 (+ 492us) server_negotiation.cc:299] Negotiation successful
0620 14:52:16.670770 (+ 494us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":723,"thread_start_us":486,"threads_started":1}
I20260620 14:52:16.677879 18690 raft_consensus.cc:348] T 00000000000000000000000000000000 P 92fc2fdb55334e25980592943ab9f8f2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "92fc2fdb55334e25980592943ab9f8f2" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 38439 } }
I20260620 14:52:16.678346 18690 raft_consensus.cc:374] T 00000000000000000000000000000000 P 92fc2fdb55334e25980592943ab9f8f2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 14:52:16.678462 18690 raft_consensus.cc:729] T 00000000000000000000000000000000 P 92fc2fdb55334e25980592943ab9f8f2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 92fc2fdb55334e25980592943ab9f8f2, State: Initialized, Role: FOLLOWER
I20260620 14:52:16.679165 18690 consensus_queue.cc:260] T 00000000000000000000000000000000 P 92fc2fdb55334e25980592943ab9f8f2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "92fc2fdb55334e25980592943ab9f8f2" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 38439 } }
I20260620 14:52:16.679453 18690 raft_consensus.cc:388] T 00000000000000000000000000000000 P 92fc2fdb55334e25980592943ab9f8f2 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 14:52:16.679694 18690 raft_consensus.cc:482] T 00000000000000000000000000000000 P 92fc2fdb55334e25980592943ab9f8f2 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 14:52:16.679940 18690 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 92fc2fdb55334e25980592943ab9f8f2 [term 0 FOLLOWER]: Advancing to term 1
I20260620 14:52:16.682657 18690 raft_consensus.cc:504] T 00000000000000000000000000000000 P 92fc2fdb55334e25980592943ab9f8f2 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "92fc2fdb55334e25980592943ab9f8f2" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 38439 } }
I20260620 14:52:16.683310 18690 leader_election.cc:302] T 00000000000000000000000000000000 P 92fc2fdb55334e25980592943ab9f8f2 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 92fc2fdb55334e25980592943ab9f8f2; no voters:
I20260620 14:52:16.684027 18690 leader_election.cc:288] T 00000000000000000000000000000000 P 92fc2fdb55334e25980592943ab9f8f2 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260620 14:52:16.684190 18695 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 92fc2fdb55334e25980592943ab9f8f2 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 14:52:16.685113 18695 raft_consensus.cc:686] T 00000000000000000000000000000000 P 92fc2fdb55334e25980592943ab9f8f2 [term 1 LEADER]: Becoming Leader. State: Replica: 92fc2fdb55334e25980592943ab9f8f2, State: Running, Role: LEADER
I20260620 14:52:16.685694 18695 consensus_queue.cc:237] T 00000000000000000000000000000000 P 92fc2fdb55334e25980592943ab9f8f2 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "92fc2fdb55334e25980592943ab9f8f2" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 38439 } }
I20260620 14:52:16.686573 18690 sys_catalog.cc:564] T 00000000000000000000000000000000 P 92fc2fdb55334e25980592943ab9f8f2 [sys.catalog]: configured and running, proceeding with master startup.
I20260620 14:52:16.689049 18696 sys_catalog.cc:455] T 00000000000000000000000000000000 P 92fc2fdb55334e25980592943ab9f8f2 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "92fc2fdb55334e25980592943ab9f8f2" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "92fc2fdb55334e25980592943ab9f8f2" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 38439 } } }
I20260620 14:52:16.689073 18697 sys_catalog.cc:455] T 00000000000000000000000000000000 P 92fc2fdb55334e25980592943ab9f8f2 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 92fc2fdb55334e25980592943ab9f8f2. Latest consensus state: current_term: 1 leader_uuid: "92fc2fdb55334e25980592943ab9f8f2" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "92fc2fdb55334e25980592943ab9f8f2" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 38439 } } }
I20260620 14:52:16.689549 18696 sys_catalog.cc:458] T 00000000000000000000000000000000 P 92fc2fdb55334e25980592943ab9f8f2 [sys.catalog]: This master's current role is: LEADER
I20260620 14:52:16.689549 18697 sys_catalog.cc:458] T 00000000000000000000000000000000 P 92fc2fdb55334e25980592943ab9f8f2 [sys.catalog]: This master's current role is: LEADER
I20260620 14:52:16.691205 18701 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 14:52:16.696943 18701 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 14:52:16.708026 18701 catalog_manager.cc:1347] Generated new cluster ID: f4d554a8171742ae8ad66604c6d0109c
I20260620 14:52:16.708169 18701 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 14:52:16.724260 18701 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 14:52:16.725680 18701 catalog_manager.cc:1504] Loading token signing keys...
I20260620 14:52:16.732836 18701 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 92fc2fdb55334e25980592943ab9f8f2: Generated new TSK 0
I20260620 14:52:16.734086 18701 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.0.97.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskUci3DU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781966829091300-388-0/krb5kdc/kudu_127.0.97.1.keytab.
Entry for principal kudu/127.0.97.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskUci3DU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781966829091300-388-0/krb5kdc/kudu_127.0.97.1.keytab.
I20260620 14:52:16.807099 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskUci3DU/build/asan/bin/kudu
/tmp/dist-test-taskUci3DU/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskUci3DU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781966829091300-388-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskUci3DU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781966829091300-388-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskUci3DU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781966829091300-388-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskUci3DU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781966829091300-388-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.1:0
--local_ip_for_outbound_sockets=127.0.97.1
--webserver_interface=127.0.97.1
--webserver_port=0
--tserver_master_addrs=127.0.97.62:38439
--keytab_file=/tmp/dist-test-taskUci3DU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781966829091300-388-0/krb5kdc/kudu_127.0.97.1.keytab
--principal=kudu/127.0.97.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:42235
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskUci3DU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781966829091300-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskUci3DU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781966829091300-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskUci3DU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781966829091300-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 14:52:17.193907 18716 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 14:52:17.194279 18716 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 14:52:17.194461 18716 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 14:52:17.202699 18716 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 14:52:17.202901 18716 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 14:52:17.203080 18716 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.1
I20260620 14:52:17.213722 18716 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:42235
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskUci3DU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781966829091300-388-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskUci3DU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781966829091300-388-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskUci3DU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781966829091300-388-0/krb5kdc/kudu_127.0.97.1.keytab
--principal=kudu/127.0.97.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskUci3DU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781966829091300-388-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.97.1
--webserver_port=0
--tserver_master_addrs=127.0.97.62:38439
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.1
--log_dir=/tmp/dist-test-taskUci3DU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781966829091300-388-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 13:41:20 UTC on ccycloud.quasar-vdagoi.root.comops.site
build id 4457462
ASAN enabled
I20260620 14:52:17.215171 18716 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 14:52:17.216596 18716 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 14:52:17.230585 18722 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 14:52:17.230659 18721 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 14:52:17.230820 18724 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 14:52:17.231889 18716 server_base.cc:1034] running on GCE node
I20260620 14:52:17.232553 18716 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 14:52:17.233526 18716 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 14:52:17.234768 18716 hybrid_clock.cc:648] HybridClock initialized: now 1781967137234707 us; error 52 us; skew 500 ppm
I20260620 14:52:17.235142 18716 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 14:52:17 dist-test-slave-q9j6 krb5kdc[18610](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781967137, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 14:52:17.238531 18716 init.cc:377] Logged in from keytab as kudu/127.0.97.1@KRBTEST.COM (short username kudu)
I20260620 14:52:17.240156 18716 webserver.cc:460] Webserver started at http://127.0.97.1:45627/ using document root <none> and password file <none>
I20260620 14:52:17.240911 18716 fs_manager.cc:359] Metadata directory not provided
I20260620 14:52:17.241061 18716 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 14:52:17.241396 18716 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 14:52:17.244678 18716 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskUci3DU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781966829091300-388-0/minicluster-data/ts-0/data/instance:
uuid: "cfcfe512d7a1477b87ee3370a4b487b5"
format_stamp: "Formatted at 2026-06-20 14:52:17 on dist-test-slave-q9j6"
server_key: "44d55f4a6cc2e4d664a0a05c9b74c6fa"
server_key_iv: "afe05d5ad3ec8fbd052fbe4ace9926be"
server_key_version: "encryptionkey@0"
I20260620 14:52:17.245576 18716 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskUci3DU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781966829091300-388-0/minicluster-data/ts-0/wal/instance:
uuid: "cfcfe512d7a1477b87ee3370a4b487b5"
format_stamp: "Formatted at 2026-06-20 14:52:17 on dist-test-slave-q9j6"
server_key: "44d55f4a6cc2e4d664a0a05c9b74c6fa"
server_key_iv: "afe05d5ad3ec8fbd052fbe4ace9926be"
server_key_version: "encryptionkey@0"
I20260620 14:52:17.251121 18716 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.000s
I20260620 14:52:17.255070 18731 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 14:52:17.256733 18716 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260620 14:52:17.256954 18716 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskUci3DU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781966829091300-388-0/minicluster-data/ts-0/data,/tmp/dist-test-taskUci3DU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781966829091300-388-0/minicluster-data/ts-0/wal
uuid: "cfcfe512d7a1477b87ee3370a4b487b5"
format_stamp: "Formatted at 2026-06-20 14:52:17 on dist-test-slave-q9j6"
server_key: "44d55f4a6cc2e4d664a0a05c9b74c6fa"
server_key_iv: "afe05d5ad3ec8fbd052fbe4ace9926be"
server_key_version: "encryptionkey@0"
I20260620 14:52:17.257201 18716 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskUci3DU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781966829091300-388-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskUci3DU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781966829091300-388-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskUci3DU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781966829091300-388-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 14:52:17.273555 18716 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 14:52:17.278405 18716 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 14:52:17.278767 18716 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 14:52:17.279989 18716 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 14:52:17.281565 18716 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 14:52:17.281682 18716 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 14:52:17.281836 18716 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 14:52:17.281912 18716 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 14:52:17.316717 18716 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.1:46621
I20260620 14:52:17.316761 18843 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.1:46621 every 8 connection(s)
I20260620 14:52:17.318073 18716 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskUci3DU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781966829091300-388-0/minicluster-data/ts-0/data/info.pb
I20260620 14:52:17.326623 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskUci3DU/build/asan/bin/kudu as pid 18716
I20260620 14:52:17.326808 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskUci3DU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781966829091300-388-0/minicluster-data/ts-0/wal/instance
I20260620 14:52:17.327346 388 external_mini_cluster.cc:1428] Setting key 6eff756046e8cefc4e8a8a76b15eecd0
Jun 20 14:52:17 dist-test-slave-q9j6 krb5kdc[18610](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781967137, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.1@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260620 14:52:17.342295 18846 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:52:17.320781 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.1:37663 (local address 127.0.97.62:38439)
0620 14:52:17.321240 (+ 459us) server_negotiation.cc:206] Beginning negotiation
0620 14:52:17.321247 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0620 14:52:17.322301 (+ 1054us) server_negotiation.cc:407] Connection header received
0620 14:52:17.324277 (+ 1976us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:52:17.324282 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:52:17.324358 (+ 76us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:52:17.324464 (+ 106us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:52:17.326053 (+ 1589us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:52:17.327220 (+ 1167us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:52:17.329175 (+ 1955us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:52:17.329687 (+ 512us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:52:17.329797 (+ 110us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:52:17.334725 (+ 4928us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 14:52:17.334736 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 14:52:17.334744 (+ 8us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 14:52:17.334799 (+ 55us) server_negotiation.cc:922] calling sasl_server_start()
0620 14:52:17.338606 (+ 3807us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:52:17.339758 (+ 1152us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:52:17.339763 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:52:17.339765 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:52:17.339896 (+ 131us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:52:17.340431 (+ 535us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:52:17.340434 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:52:17.340436 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:52:17.340718 (+ 282us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 14:52:17.340836 (+ 118us) server_negotiation.cc:1035] Waiting for connection context
0620 14:52:17.341836 (+ 1000us) server_negotiation.cc:299] Negotiation successful
0620 14:52:17.342023 (+ 187us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":333,"thread_start_us":217,"threads_started":1}
I20260620 14:52:17.343991 18847 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:52:17.321391 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:38439 (local address 127.0.97.1:37663)
0620 14:52:17.322117 (+ 726us) negotiation.cc:107] Waiting for socket to connect
0620 14:52:17.322164 (+ 47us) client_negotiation.cc:174] Beginning negotiation
0620 14:52:17.323952 (+ 1788us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 14:52:17.324687 (+ 735us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 14:52:17.324696 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 14:52:17.325254 (+ 558us) client_negotiation.cc:189] Negotiated authn=SASL
0620 14:52:17.325850 (+ 596us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:52:17.325867 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:52:17.327398 (+ 1531us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:52:17.327402 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:52:17.328976 (+ 1574us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:52:17.328989 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:52:17.329940 (+ 951us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:52:17.329948 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:52:17.330287 (+ 339us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:52:17.331076 (+ 789us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 14:52:17.331114 (+ 38us) client_negotiation.cc:656] calling sasl_client_start()
0620 14:52:17.334394 (+ 3280us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 14:52:17.338818 (+ 4424us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 14:52:17.338831 (+ 13us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 14:52:17.338855 (+ 24us) client_negotiation.cc:760] calling sasl_client_step()
0620 14:52:17.339630 (+ 775us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 14:52:17.340016 (+ 386us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 14:52:17.340023 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 14:52:17.340030 (+ 7us) client_negotiation.cc:760] calling sasl_client_step()
0620 14:52:17.340310 (+ 280us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 14:52:17.340853 (+ 543us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 14:52:17.340866 (+ 13us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 14:52:17.341386 (+ 520us) client_negotiation.cc:769] Sending connection context
0620 14:52:17.341735 (+ 349us) client_negotiation.cc:240] Negotiation successful
0620 14:52:17.342377 (+ 642us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":479,"thread_start_us":273,"threads_started":1}
I20260620 14:52:17.345767 18844 heartbeater.cc:347] Connected to a master server at 127.0.97.62:38439
I20260620 14:52:17.346115 18844 heartbeater.cc:464] Registering TS with master...
I20260620 14:52:17.346905 18844 heartbeater.cc:511] Master 127.0.97.62:38439 requested a full tablet report, sending...
I20260620 14:52:17.349166 18655 ts_manager.cc:194] Registered new tserver with Master: cfcfe512d7a1477b87ee3370a4b487b5 (127.0.97.1:46621)
I20260620 14:52:17.350716 18655 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.97.1@KRBTEST.COM'} at 127.0.97.1:37663
WARNING: no policy specified for kudu/127.0.97.2@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskUci3DU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781966829091300-388-0/krb5kdc/kudu_127.0.97.2.keytab.
Entry for principal kudu/127.0.97.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskUci3DU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781966829091300-388-0/krb5kdc/kudu_127.0.97.2.keytab.
I20260620 14:52:17.386621 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskUci3DU/build/asan/bin/kudu
/tmp/dist-test-taskUci3DU/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskUci3DU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781966829091300-388-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskUci3DU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781966829091300-388-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskUci3DU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781966829091300-388-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskUci3DU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781966829091300-388-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.2:0
--local_ip_for_outbound_sockets=127.0.97.2
--webserver_interface=127.0.97.2
--webserver_port=0
--tserver_master_addrs=127.0.97.62:38439
--keytab_file=/tmp/dist-test-taskUci3DU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781966829091300-388-0/krb5kdc/kudu_127.0.97.2.keytab
--principal=kudu/127.0.97.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:42235
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskUci3DU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781966829091300-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskUci3DU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781966829091300-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskUci3DU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781966829091300-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 14:52:17.732918 18850 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 14:52:17.733269 18850 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 14:52:17.733441 18850 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 14:52:17.741061 18850 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 14:52:17.741227 18850 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 14:52:17.741398 18850 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.2
I20260620 14:52:17.751749 18850 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:42235
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskUci3DU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781966829091300-388-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskUci3DU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781966829091300-388-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskUci3DU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781966829091300-388-0/krb5kdc/kudu_127.0.97.2.keytab
--principal=kudu/127.0.97.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskUci3DU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781966829091300-388-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.97.2
--webserver_port=0
--tserver_master_addrs=127.0.97.62:38439
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.2
--log_dir=/tmp/dist-test-taskUci3DU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781966829091300-388-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 13:41:20 UTC on ccycloud.quasar-vdagoi.root.comops.site
build id 4457462
ASAN enabled
I20260620 14:52:17.753162 18850 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 14:52:17.754637 18850 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 14:52:17.769179 18855 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 14:52:17.769161 18858 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 14:52:17.769201 18856 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 14:52:17.770123 18850 server_base.cc:1034] running on GCE node
I20260620 14:52:17.770747 18850 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 14:52:17.771770 18850 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 14:52:17.772989 18850 hybrid_clock.cc:648] HybridClock initialized: now 1781967137772931 us; error 52 us; skew 500 ppm
I20260620 14:52:17.773358 18850 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 14:52:17 dist-test-slave-q9j6 krb5kdc[18610](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781967137, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 14:52:17.776815 18850 init.cc:377] Logged in from keytab as kudu/127.0.97.2@KRBTEST.COM (short username kudu)
I20260620 14:52:17.778452 18850 webserver.cc:460] Webserver started at http://127.0.97.2:36767/ using document root <none> and password file <none>
I20260620 14:52:17.779284 18850 fs_manager.cc:359] Metadata directory not provided
I20260620 14:52:17.779423 18850 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 14:52:17.779839 18850 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 14:52:17.783120 18850 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskUci3DU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781966829091300-388-0/minicluster-data/ts-1/data/instance:
uuid: "8ff43444b87a45ec97617f5fe374dad4"
format_stamp: "Formatted at 2026-06-20 14:52:17 on dist-test-slave-q9j6"
server_key: "c435c6e41ec65e516ebaf473516a9b96"
server_key_iv: "0e10c351d6eafa57ca7267bc641e8169"
server_key_version: "encryptionkey@0"
I20260620 14:52:17.784020 18850 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskUci3DU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781966829091300-388-0/minicluster-data/ts-1/wal/instance:
uuid: "8ff43444b87a45ec97617f5fe374dad4"
format_stamp: "Formatted at 2026-06-20 14:52:17 on dist-test-slave-q9j6"
server_key: "c435c6e41ec65e516ebaf473516a9b96"
server_key_iv: "0e10c351d6eafa57ca7267bc641e8169"
server_key_version: "encryptionkey@0"
I20260620 14:52:17.789995 18850 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.005s sys 0.000s
I20260620 14:52:17.793936 18865 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 14:52:17.795549 18850 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260620 14:52:17.795825 18850 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskUci3DU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781966829091300-388-0/minicluster-data/ts-1/data,/tmp/dist-test-taskUci3DU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781966829091300-388-0/minicluster-data/ts-1/wal
uuid: "8ff43444b87a45ec97617f5fe374dad4"
format_stamp: "Formatted at 2026-06-20 14:52:17 on dist-test-slave-q9j6"
server_key: "c435c6e41ec65e516ebaf473516a9b96"
server_key_iv: "0e10c351d6eafa57ca7267bc641e8169"
server_key_version: "encryptionkey@0"
I20260620 14:52:17.796101 18850 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskUci3DU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781966829091300-388-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskUci3DU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781966829091300-388-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskUci3DU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781966829091300-388-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 14:52:17.818413 18850 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 14:52:17.823369 18850 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 14:52:17.823752 18850 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 14:52:17.824867 18850 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 14:52:17.826328 18850 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 14:52:17.826407 18850 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 14:52:17.826603 18850 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 14:52:17.826656 18850 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 14:52:17.861269 18850 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.2:42185
I20260620 14:52:17.861290 18977 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.2:42185 every 8 connection(s)
I20260620 14:52:17.862782 18850 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskUci3DU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781966829091300-388-0/minicluster-data/ts-1/data/info.pb
I20260620 14:52:17.863257 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskUci3DU/build/asan/bin/kudu as pid 18850
I20260620 14:52:17.863451 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskUci3DU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781966829091300-388-0/minicluster-data/ts-1/wal/instance
I20260620 14:52:17.864104 388 external_mini_cluster.cc:1428] Setting key ee1fecce34ec747b4490de597b40b1bc
Jun 20 14:52:17 dist-test-slave-q9j6 krb5kdc[18610](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781967137, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.2@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260620 14:52:17.885377 18980 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:52:17.865568 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.2:52499 (local address 127.0.97.62:38439)
0620 14:52:17.866069 (+ 501us) server_negotiation.cc:206] Beginning negotiation
0620 14:52:17.866075 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 14:52:17.867424 (+ 1349us) server_negotiation.cc:407] Connection header received
0620 14:52:17.869414 (+ 1990us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:52:17.869422 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:52:17.869547 (+ 125us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:52:17.869672 (+ 125us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:52:17.871276 (+ 1604us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:52:17.872509 (+ 1233us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:52:17.874314 (+ 1805us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:52:17.874755 (+ 441us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:52:17.874905 (+ 150us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:52:17.878871 (+ 3966us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 14:52:17.878877 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 14:52:17.878880 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 14:52:17.878917 (+ 37us) server_negotiation.cc:922] calling sasl_server_start()
0620 14:52:17.882001 (+ 3084us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:52:17.882831 (+ 830us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:52:17.882838 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:52:17.882841 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:52:17.882995 (+ 154us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:52:17.883449 (+ 454us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:52:17.883454 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:52:17.883458 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:52:17.883817 (+ 359us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 14:52:17.883999 (+ 182us) server_negotiation.cc:1035] Waiting for connection context
0620 14:52:17.884735 (+ 736us) server_negotiation.cc:299] Negotiation successful
0620 14:52:17.884991 (+ 256us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":352,"thread_start_us":246,"threads_started":1}
I20260620 14:52:17.886584 18981 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:52:17.866314 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:38439 (local address 127.0.97.2:52499)
0620 14:52:17.867220 (+ 906us) negotiation.cc:107] Waiting for socket to connect
0620 14:52:17.867276 (+ 56us) client_negotiation.cc:174] Beginning negotiation
0620 14:52:17.869104 (+ 1828us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 14:52:17.869900 (+ 796us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 14:52:17.869908 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 14:52:17.870455 (+ 547us) client_negotiation.cc:189] Negotiated authn=SASL
0620 14:52:17.871060 (+ 605us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:52:17.871078 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:52:17.872705 (+ 1627us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:52:17.872709 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:52:17.874038 (+ 1329us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:52:17.874052 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:52:17.875016 (+ 964us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:52:17.875024 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:52:17.875273 (+ 249us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:52:17.876067 (+ 794us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 14:52:17.876096 (+ 29us) client_negotiation.cc:656] calling sasl_client_start()
0620 14:52:17.878591 (+ 2495us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 14:52:17.882178 (+ 3587us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 14:52:17.882187 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 14:52:17.882207 (+ 20us) client_negotiation.cc:760] calling sasl_client_step()
0620 14:52:17.882694 (+ 487us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 14:52:17.883114 (+ 420us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 14:52:17.883119 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 14:52:17.883123 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0620 14:52:17.883333 (+ 210us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 14:52:17.883949 (+ 616us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 14:52:17.883958 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 14:52:17.884355 (+ 397us) client_negotiation.cc:769] Sending connection context
0620 14:52:17.884656 (+ 301us) client_negotiation.cc:240] Negotiation successful
0620 14:52:17.885250 (+ 594us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":618,"thread_start_us":362,"threads_started":1}
I20260620 14:52:17.888258 18978 heartbeater.cc:347] Connected to a master server at 127.0.97.62:38439
I20260620 14:52:17.888669 18978 heartbeater.cc:464] Registering TS with master...
WARNING: no policy specified for kudu/127.0.97.3@KRBTEST.COM; defaulting to no policy
I20260620 14:52:17.889680 18978 heartbeater.cc:511] Master 127.0.97.62:38439 requested a full tablet report, sending...
I20260620 14:52:17.891431 18655 ts_manager.cc:194] Registered new tserver with Master: 8ff43444b87a45ec97617f5fe374dad4 (127.0.97.2:42185)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.3@KRBTEST.COM" created.
I20260620 14:52:17.892614 18655 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.97.2@KRBTEST.COM'} at 127.0.97.2:52499
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskUci3DU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781966829091300-388-0/krb5kdc/kudu_127.0.97.3.keytab.
Entry for principal kudu/127.0.97.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskUci3DU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781966829091300-388-0/krb5kdc/kudu_127.0.97.3.keytab.
I20260620 14:52:17.922446 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskUci3DU/build/asan/bin/kudu
/tmp/dist-test-taskUci3DU/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskUci3DU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781966829091300-388-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskUci3DU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781966829091300-388-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskUci3DU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781966829091300-388-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskUci3DU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781966829091300-388-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.3:0
--local_ip_for_outbound_sockets=127.0.97.3
--webserver_interface=127.0.97.3
--webserver_port=0
--tserver_master_addrs=127.0.97.62:38439
--keytab_file=/tmp/dist-test-taskUci3DU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781966829091300-388-0/krb5kdc/kudu_127.0.97.3.keytab
--principal=kudu/127.0.97.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:42235
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskUci3DU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781966829091300-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskUci3DU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781966829091300-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskUci3DU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781966829091300-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 14:52:18.271247 18984 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 14:52:18.271579 18984 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 14:52:18.271711 18984 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 14:52:18.279532 18984 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 14:52:18.279691 18984 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 14:52:18.279878 18984 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.3
I20260620 14:52:18.290405 18984 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:42235
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskUci3DU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781966829091300-388-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskUci3DU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781966829091300-388-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskUci3DU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781966829091300-388-0/krb5kdc/kudu_127.0.97.3.keytab
--principal=kudu/127.0.97.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskUci3DU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781966829091300-388-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.97.3
--webserver_port=0
--tserver_master_addrs=127.0.97.62:38439
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.3
--log_dir=/tmp/dist-test-taskUci3DU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781966829091300-388-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 13:41:20 UTC on ccycloud.quasar-vdagoi.root.comops.site
build id 4457462
ASAN enabled
I20260620 14:52:18.291890 18984 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 14:52:18.293430 18984 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 14:52:18.308032 18989 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 14:52:18.308068 18992 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 14:52:18.308004 18990 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 14:52:18.309396 18984 server_base.cc:1034] running on GCE node
I20260620 14:52:18.310087 18984 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 14:52:18.311136 18984 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 14:52:18.312359 18984 hybrid_clock.cc:648] HybridClock initialized: now 1781967138312311 us; error 38 us; skew 500 ppm
I20260620 14:52:18.312728 18984 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 14:52:18 dist-test-slave-q9j6 krb5kdc[18610](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781967138, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 14:52:18.316263 18984 init.cc:377] Logged in from keytab as kudu/127.0.97.3@KRBTEST.COM (short username kudu)
I20260620 14:52:18.318194 18984 webserver.cc:460] Webserver started at http://127.0.97.3:34597/ using document root <none> and password file <none>
I20260620 14:52:18.319187 18984 fs_manager.cc:359] Metadata directory not provided
I20260620 14:52:18.319335 18984 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 14:52:18.319739 18984 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 14:52:18.323873 18984 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskUci3DU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781966829091300-388-0/minicluster-data/ts-2/data/instance:
uuid: "07a75a6c306d4a03acc972cccb741e6e"
format_stamp: "Formatted at 2026-06-20 14:52:18 on dist-test-slave-q9j6"
server_key: "1e9858a225b2d9c936607761f125c841"
server_key_iv: "4991c27f43883903150c6aa02161c904"
server_key_version: "encryptionkey@0"
I20260620 14:52:18.324672 18984 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskUci3DU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781966829091300-388-0/minicluster-data/ts-2/wal/instance:
uuid: "07a75a6c306d4a03acc972cccb741e6e"
format_stamp: "Formatted at 2026-06-20 14:52:18 on dist-test-slave-q9j6"
server_key: "1e9858a225b2d9c936607761f125c841"
server_key_iv: "4991c27f43883903150c6aa02161c904"
server_key_version: "encryptionkey@0"
I20260620 14:52:18.330206 18984 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260620 14:52:18.334162 18999 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 14:52:18.335791 18984 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260620 14:52:18.335990 18984 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskUci3DU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781966829091300-388-0/minicluster-data/ts-2/data,/tmp/dist-test-taskUci3DU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781966829091300-388-0/minicluster-data/ts-2/wal
uuid: "07a75a6c306d4a03acc972cccb741e6e"
format_stamp: "Formatted at 2026-06-20 14:52:18 on dist-test-slave-q9j6"
server_key: "1e9858a225b2d9c936607761f125c841"
server_key_iv: "4991c27f43883903150c6aa02161c904"
server_key_version: "encryptionkey@0"
I20260620 14:52:18.336231 18984 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskUci3DU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781966829091300-388-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskUci3DU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781966829091300-388-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskUci3DU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781966829091300-388-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 14:52:18.354136 18844 heartbeater.cc:503] Master 127.0.97.62:38439 was elected leader, sending a full tablet report...
I20260620 14:52:18.359367 18984 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 14:52:18.364830 18984 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 14:52:18.365183 18984 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 14:52:18.366313 18984 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 14:52:18.367851 18984 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 14:52:18.367933 18984 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 14:52:18.368098 18984 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 14:52:18.368150 18984 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 14:52:18.404003 18984 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.3:39011
I20260620 14:52:18.404059 19111 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.3:39011 every 8 connection(s)
I20260620 14:52:18.405383 18984 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskUci3DU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781966829091300-388-0/minicluster-data/ts-2/data/info.pb
I20260620 14:52:18.408154 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskUci3DU/build/asan/bin/kudu as pid 18984
I20260620 14:52:18.408358 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskUci3DU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781966829091300-388-0/minicluster-data/ts-2/wal/instance
I20260620 14:52:18.408882 388 external_mini_cluster.cc:1428] Setting key 34b272880f98f3e31c4a5d4bdb0fe26b
Jun 20 14:52:18 dist-test-slave-q9j6 krb5kdc[18610](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781967138, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.3@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260620 14:52:18.428061 19115 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:52:18.409462 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.3:33427 (local address 127.0.97.62:38439)
0620 14:52:18.409947 (+ 485us) server_negotiation.cc:206] Beginning negotiation
0620 14:52:18.409953 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 14:52:18.410075 (+ 122us) server_negotiation.cc:407] Connection header received
0620 14:52:18.412725 (+ 2650us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:52:18.412729 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:52:18.412815 (+ 86us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:52:18.412911 (+ 96us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:52:18.414584 (+ 1673us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:52:18.415477 (+ 893us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:52:18.417126 (+ 1649us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:52:18.417542 (+ 416us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:52:18.417643 (+ 101us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:52:18.421328 (+ 3685us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 14:52:18.421333 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 14:52:18.421337 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 14:52:18.421375 (+ 38us) server_negotiation.cc:922] calling sasl_server_start()
0620 14:52:18.424686 (+ 3311us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:52:18.425615 (+ 929us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:52:18.425620 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:52:18.425622 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:52:18.425753 (+ 131us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:52:18.426168 (+ 415us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:52:18.426172 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:52:18.426174 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:52:18.426443 (+ 269us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 14:52:18.426608 (+ 165us) server_negotiation.cc:1035] Waiting for connection context
0620 14:52:18.427363 (+ 755us) server_negotiation.cc:299] Negotiation successful
0620 14:52:18.427634 (+ 271us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":357,"thread_start_us":1054,"threads_started":1}
I20260620 14:52:18.429164 19114 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:52:18.408903 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:38439 (local address 127.0.97.3:33427)
0620 14:52:18.409901 (+ 998us) negotiation.cc:107] Waiting for socket to connect
0620 14:52:18.409951 (+ 50us) client_negotiation.cc:174] Beginning negotiation
0620 14:52:18.412441 (+ 2490us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 14:52:18.413168 (+ 727us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 14:52:18.413177 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 14:52:18.413735 (+ 558us) client_negotiation.cc:189] Negotiated authn=SASL
0620 14:52:18.414382 (+ 647us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:52:18.414405 (+ 23us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:52:18.415684 (+ 1279us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:52:18.415688 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:52:18.416977 (+ 1289us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:52:18.416990 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:52:18.417744 (+ 754us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:52:18.417748 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:52:18.417959 (+ 211us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:52:18.418733 (+ 774us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 14:52:18.418758 (+ 25us) client_negotiation.cc:656] calling sasl_client_start()
0620 14:52:18.421081 (+ 2323us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 14:52:18.424885 (+ 3804us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 14:52:18.424899 (+ 14us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 14:52:18.424921 (+ 22us) client_negotiation.cc:760] calling sasl_client_step()
0620 14:52:18.425485 (+ 564us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 14:52:18.425856 (+ 371us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 14:52:18.425859 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 14:52:18.425861 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 14:52:18.426068 (+ 207us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 14:52:18.426584 (+ 516us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 14:52:18.426594 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 14:52:18.426972 (+ 378us) client_negotiation.cc:769] Sending connection context
0620 14:52:18.427258 (+ 286us) client_negotiation.cc:240] Negotiation successful
0620 14:52:18.427887 (+ 629us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":639,"thread_start_us":584,"threads_started":1}
I20260620 14:52:18.430680 19112 heartbeater.cc:347] Connected to a master server at 127.0.97.62:38439
I20260620 14:52:18.431027 19112 heartbeater.cc:464] Registering TS with master...
I20260620 14:52:18.431872 19112 heartbeater.cc:511] Master 127.0.97.62:38439 requested a full tablet report, sending...
I20260620 14:52:18.433507 18655 ts_manager.cc:194] Registered new tserver with Master: 07a75a6c306d4a03acc972cccb741e6e (127.0.97.3:39011)
I20260620 14:52:18.434373 18655 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.97.3@KRBTEST.COM'} at 127.0.97.3:33427
I20260620 14:52:18.437709 388 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260620 14:52:18.456215 19115 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:52:18.442219 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44524 (local address 127.0.97.62:38439)
0620 14:52:18.442403 (+ 184us) server_negotiation.cc:206] Beginning negotiation
0620 14:52:18.442408 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 14:52:18.442943 (+ 535us) server_negotiation.cc:407] Connection header received
0620 14:52:18.443089 (+ 146us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:52:18.443094 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:52:18.443164 (+ 70us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:52:18.443257 (+ 93us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:52:18.444857 (+ 1600us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:52:18.445633 (+ 776us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:52:18.447118 (+ 1485us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:52:18.447540 (+ 422us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:52:18.447744 (+ 204us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:52:18.450529 (+ 2785us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 14:52:18.450534 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 14:52:18.450538 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 14:52:18.450576 (+ 38us) server_negotiation.cc:922] calling sasl_server_start()
0620 14:52:18.453411 (+ 2835us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:52:18.454355 (+ 944us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:52:18.454359 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:52:18.454361 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:52:18.454488 (+ 127us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:52:18.454892 (+ 404us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:52:18.454895 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:52:18.454897 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:52:18.455229 (+ 332us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 14:52:18.455435 (+ 206us) server_negotiation.cc:1035] Waiting for connection context
0620 14:52:18.455811 (+ 376us) server_negotiation.cc:299] Negotiation successful
0620 14:52:18.455970 (+ 159us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":69}
I20260620 14:52:18.460819 18655 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:44524:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260620 14:52:18.467999 18655 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260620 14:52:18.509178 19127 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:52:18.481410 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34114 (local address 127.0.97.2:42185)
0620 14:52:18.498438 (+ 17028us) server_negotiation.cc:206] Beginning negotiation
0620 14:52:18.498452 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0620 14:52:18.498499 (+ 47us) server_negotiation.cc:407] Connection header received
0620 14:52:18.498640 (+ 141us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:52:18.498653 (+ 13us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:52:18.499045 (+ 392us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:52:18.499330 (+ 285us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 14:52:18.502282 (+ 2952us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:52:18.503878 (+ 1596us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:52:18.506351 (+ 2473us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:52:18.507925 (+ 1574us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:52:18.508079 (+ 154us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:52:18.508311 (+ 232us) server_negotiation.cc:1035] Waiting for connection context
0620 14:52:18.508626 (+ 315us) server_negotiation.cc:299] Negotiation successful
0620 14:52:18.508801 (+ 175us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":16446,"spinlock_wait_cycles":528640,"thread_start_us":16513,"threads_started":1}
I20260620 14:52:18.509212 19129 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:52:18.481200 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.2:42185 (local address 127.0.0.1:34114)
0620 14:52:18.497017 (+ 15817us) negotiation.cc:107] Waiting for socket to connect
0620 14:52:18.497045 (+ 28us) client_negotiation.cc:174] Beginning negotiation
0620 14:52:18.497310 (+ 265us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 14:52:18.499750 (+ 2440us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 14:52:18.499762 (+ 12us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 14:52:18.499826 (+ 64us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 14:52:18.501662 (+ 1836us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:52:18.501678 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:52:18.504009 (+ 2331us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:52:18.504017 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:52:18.506121 (+ 2104us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:52:18.506144 (+ 23us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:52:18.508121 (+ 1977us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:52:18.508127 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:52:18.508344 (+ 217us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:52:18.508408 (+ 64us) client_negotiation.cc:769] Sending connection context
0620 14:52:18.508574 (+ 166us) client_negotiation.cc:240] Negotiation successful
0620 14:52:18.508784 (+ 210us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":15650,"mutex_wait_us":27,"thread_start_us":19686,"threads_started":1}
I20260620 14:52:18.510257 19126 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:52:18.484584 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:46621 (local address 127.0.0.1:55892)
0620 14:52:18.501300 (+ 16716us) negotiation.cc:107] Waiting for socket to connect
0620 14:52:18.501326 (+ 26us) client_negotiation.cc:174] Beginning negotiation
0620 14:52:18.501518 (+ 192us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 14:52:18.502401 (+ 883us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 14:52:18.502405 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 14:52:18.502431 (+ 26us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 14:52:18.502842 (+ 411us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:52:18.502859 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:52:18.504940 (+ 2081us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:52:18.504947 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:52:18.507055 (+ 2108us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:52:18.507072 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:52:18.509343 (+ 2271us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:52:18.509352 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:52:18.509560 (+ 208us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:52:18.509591 (+ 31us) client_negotiation.cc:769] Sending connection context
0620 14:52:18.509674 (+ 83us) client_negotiation.cc:240] Negotiation successful
0620 14:52:18.509833 (+ 159us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":16524,"mutex_wait_us":4,"thread_start_us":12020,"threads_started":1}
I20260620 14:52:18.510633 19130 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:52:18.485345 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55892 (local address 127.0.97.1:46621)
0620 14:52:18.496441 (+ 11096us) server_negotiation.cc:206] Beginning negotiation
0620 14:52:18.496455 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0620 14:52:18.501720 (+ 5265us) server_negotiation.cc:407] Connection header received
0620 14:52:18.501866 (+ 146us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:52:18.501878 (+ 12us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:52:18.502202 (+ 324us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:52:18.502429 (+ 227us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 14:52:18.503080 (+ 651us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:52:18.504768 (+ 1688us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:52:18.507276 (+ 2508us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:52:18.508788 (+ 1512us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:52:18.508921 (+ 133us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:52:18.509170 (+ 249us) server_negotiation.cc:1035] Waiting for connection context
0620 14:52:18.509897 (+ 727us) server_negotiation.cc:299] Negotiation successful
0620 14:52:18.510079 (+ 182us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":10859,"thread_start_us":10622,"threads_started":1}
I20260620 14:52:18.510656 19125 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:52:18.480938 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.3:39011 (local address 127.0.0.1:55768)
0620 14:52:18.493442 (+ 12504us) negotiation.cc:107] Waiting for socket to connect
0620 14:52:18.493508 (+ 66us) client_negotiation.cc:174] Beginning negotiation
0620 14:52:18.493767 (+ 259us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 14:52:18.502931 (+ 9164us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 14:52:18.502937 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 14:52:18.502963 (+ 26us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 14:52:18.503262 (+ 299us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:52:18.503272 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:52:18.505169 (+ 1897us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:52:18.505175 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:52:18.507571 (+ 2396us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:52:18.507588 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:52:18.509843 (+ 2255us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:52:18.509849 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:52:18.510012 (+ 163us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:52:18.510043 (+ 31us) client_negotiation.cc:769] Sending connection context
0620 14:52:18.510138 (+ 95us) client_negotiation.cc:240] Negotiation successful
0620 14:52:18.510266 (+ 128us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":12114,"mutex_wait_us":65,"thread_start_us":13079,"threads_started":1}
I20260620 14:52:18.511389 19128 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:52:18.481262 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55768 (local address 127.0.97.3:39011)
0620 14:52:18.502289 (+ 21027us) server_negotiation.cc:206] Beginning negotiation
0620 14:52:18.502297 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0620 14:52:18.502328 (+ 31us) server_negotiation.cc:407] Connection header received
0620 14:52:18.502418 (+ 90us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:52:18.502425 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:52:18.502692 (+ 267us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:52:18.502905 (+ 213us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 14:52:18.503454 (+ 549us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:52:18.505000 (+ 1546us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:52:18.507807 (+ 2807us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:52:18.509254 (+ 1447us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:52:18.509542 (+ 288us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:52:18.509784 (+ 242us) server_negotiation.cc:1035] Waiting for connection context
0620 14:52:18.510880 (+ 1096us) server_negotiation.cc:299] Negotiation successful
0620 14:52:18.511062 (+ 182us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":20859,"thread_start_us":20711,"threads_started":1}
I20260620 14:52:18.514485 19047 tablet_service.cc:1467] Processing CreateTablet for tablet 23028294819a44e2926efbf60cf334de (DEFAULT_TABLE table=test-table [id=41b053796bd14d5ab5db998935c1f70b]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 14:52:18.514509 18913 tablet_service.cc:1467] Processing CreateTablet for tablet 23028294819a44e2926efbf60cf334de (DEFAULT_TABLE table=test-table [id=41b053796bd14d5ab5db998935c1f70b]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 14:52:18.514506 18779 tablet_service.cc:1467] Processing CreateTablet for tablet 23028294819a44e2926efbf60cf334de (DEFAULT_TABLE table=test-table [id=41b053796bd14d5ab5db998935c1f70b]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 14:52:18.516009 18913 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 23028294819a44e2926efbf60cf334de. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 14:52:18.516010 18779 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 23028294819a44e2926efbf60cf334de. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 14:52:18.516006 19047 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 23028294819a44e2926efbf60cf334de. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 14:52:18.528275 19131 tablet_bootstrap.cc:492] T 23028294819a44e2926efbf60cf334de P 07a75a6c306d4a03acc972cccb741e6e: Bootstrap starting.
I20260620 14:52:18.531373 19132 tablet_bootstrap.cc:492] T 23028294819a44e2926efbf60cf334de P cfcfe512d7a1477b87ee3370a4b487b5: Bootstrap starting.
I20260620 14:52:18.532131 19131 tablet_bootstrap.cc:654] T 23028294819a44e2926efbf60cf334de P 07a75a6c306d4a03acc972cccb741e6e: Neither blocks nor log segments found. Creating new log.
I20260620 14:52:18.533584 19131 log.cc:824] T 23028294819a44e2926efbf60cf334de P 07a75a6c306d4a03acc972cccb741e6e: Log is configured to *not* fsync() on all Append() calls
I20260620 14:52:18.535202 19132 tablet_bootstrap.cc:654] T 23028294819a44e2926efbf60cf334de P cfcfe512d7a1477b87ee3370a4b487b5: Neither blocks nor log segments found. Creating new log.
I20260620 14:52:18.535180 19133 tablet_bootstrap.cc:492] T 23028294819a44e2926efbf60cf334de P 8ff43444b87a45ec97617f5fe374dad4: Bootstrap starting.
I20260620 14:52:18.536561 19132 log.cc:824] T 23028294819a44e2926efbf60cf334de P cfcfe512d7a1477b87ee3370a4b487b5: Log is configured to *not* fsync() on all Append() calls
I20260620 14:52:18.538520 19131 tablet_bootstrap.cc:492] T 23028294819a44e2926efbf60cf334de P 07a75a6c306d4a03acc972cccb741e6e: No bootstrap required, opened a new log
I20260620 14:52:18.538792 19133 tablet_bootstrap.cc:654] T 23028294819a44e2926efbf60cf334de P 8ff43444b87a45ec97617f5fe374dad4: Neither blocks nor log segments found. Creating new log.
I20260620 14:52:18.538890 19131 ts_tablet_manager.cc:1397] T 23028294819a44e2926efbf60cf334de P 07a75a6c306d4a03acc972cccb741e6e: Time spent bootstrapping tablet: real 0.011s user 0.006s sys 0.000s
I20260620 14:52:18.539986 19132 tablet_bootstrap.cc:492] T 23028294819a44e2926efbf60cf334de P cfcfe512d7a1477b87ee3370a4b487b5: No bootstrap required, opened a new log
I20260620 14:52:18.540165 19133 log.cc:824] T 23028294819a44e2926efbf60cf334de P 8ff43444b87a45ec97617f5fe374dad4: Log is configured to *not* fsync() on all Append() calls
I20260620 14:52:18.540390 19132 ts_tablet_manager.cc:1397] T 23028294819a44e2926efbf60cf334de P cfcfe512d7a1477b87ee3370a4b487b5: Time spent bootstrapping tablet: real 0.009s user 0.006s sys 0.003s
I20260620 14:52:18.543663 19133 tablet_bootstrap.cc:492] T 23028294819a44e2926efbf60cf334de P 8ff43444b87a45ec97617f5fe374dad4: No bootstrap required, opened a new log
I20260620 14:52:18.543963 19133 ts_tablet_manager.cc:1397] T 23028294819a44e2926efbf60cf334de P 8ff43444b87a45ec97617f5fe374dad4: Time spent bootstrapping tablet: real 0.009s user 0.008s sys 0.000s
I20260620 14:52:18.547294 19131 raft_consensus.cc:348] T 23028294819a44e2926efbf60cf334de P 07a75a6c306d4a03acc972cccb741e6e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "07a75a6c306d4a03acc972cccb741e6e" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 39011 } } peers { permanent_uuid: "8ff43444b87a45ec97617f5fe374dad4" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 42185 } } peers { permanent_uuid: "cfcfe512d7a1477b87ee3370a4b487b5" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 46621 } }
I20260620 14:52:18.547744 19131 raft_consensus.cc:374] T 23028294819a44e2926efbf60cf334de P 07a75a6c306d4a03acc972cccb741e6e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 14:52:18.547838 19131 raft_consensus.cc:729] T 23028294819a44e2926efbf60cf334de P 07a75a6c306d4a03acc972cccb741e6e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 07a75a6c306d4a03acc972cccb741e6e, State: Initialized, Role: FOLLOWER
I20260620 14:52:18.548520 19131 consensus_queue.cc:260] T 23028294819a44e2926efbf60cf334de P 07a75a6c306d4a03acc972cccb741e6e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "07a75a6c306d4a03acc972cccb741e6e" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 39011 } } peers { permanent_uuid: "8ff43444b87a45ec97617f5fe374dad4" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 42185 } } peers { permanent_uuid: "cfcfe512d7a1477b87ee3370a4b487b5" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 46621 } }
I20260620 14:52:18.551345 19132 raft_consensus.cc:348] T 23028294819a44e2926efbf60cf334de P cfcfe512d7a1477b87ee3370a4b487b5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "07a75a6c306d4a03acc972cccb741e6e" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 39011 } } peers { permanent_uuid: "8ff43444b87a45ec97617f5fe374dad4" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 42185 } } peers { permanent_uuid: "cfcfe512d7a1477b87ee3370a4b487b5" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 46621 } }
I20260620 14:52:18.551867 19132 raft_consensus.cc:374] T 23028294819a44e2926efbf60cf334de P cfcfe512d7a1477b87ee3370a4b487b5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 14:52:18.551966 19132 raft_consensus.cc:729] T 23028294819a44e2926efbf60cf334de P cfcfe512d7a1477b87ee3370a4b487b5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: cfcfe512d7a1477b87ee3370a4b487b5, State: Initialized, Role: FOLLOWER
I20260620 14:52:18.552744 19132 consensus_queue.cc:260] T 23028294819a44e2926efbf60cf334de P cfcfe512d7a1477b87ee3370a4b487b5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "07a75a6c306d4a03acc972cccb741e6e" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 39011 } } peers { permanent_uuid: "8ff43444b87a45ec97617f5fe374dad4" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 42185 } } peers { permanent_uuid: "cfcfe512d7a1477b87ee3370a4b487b5" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 46621 } }
I20260620 14:52:18.556514 19138 raft_consensus.cc:482] T 23028294819a44e2926efbf60cf334de P 07a75a6c306d4a03acc972cccb741e6e [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260620 14:52:18.556704 19112 heartbeater.cc:503] Master 127.0.97.62:38439 was elected leader, sending a full tablet report...
I20260620 14:52:18.556902 19138 raft_consensus.cc:504] T 23028294819a44e2926efbf60cf334de P 07a75a6c306d4a03acc972cccb741e6e [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "07a75a6c306d4a03acc972cccb741e6e" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 39011 } } peers { permanent_uuid: "8ff43444b87a45ec97617f5fe374dad4" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 42185 } } peers { permanent_uuid: "cfcfe512d7a1477b87ee3370a4b487b5" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 46621 } }
I20260620 14:52:18.561722 19132 ts_tablet_manager.cc:1428] T 23028294819a44e2926efbf60cf334de P cfcfe512d7a1477b87ee3370a4b487b5: Time spent starting tablet: real 0.021s user 0.019s sys 0.001s
I20260620 14:52:18.555455 19133 raft_consensus.cc:348] T 23028294819a44e2926efbf60cf334de P 8ff43444b87a45ec97617f5fe374dad4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "07a75a6c306d4a03acc972cccb741e6e" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 39011 } } peers { permanent_uuid: "8ff43444b87a45ec97617f5fe374dad4" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 42185 } } peers { permanent_uuid: "cfcfe512d7a1477b87ee3370a4b487b5" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 46621 } }
I20260620 14:52:18.565804 19133 raft_consensus.cc:374] T 23028294819a44e2926efbf60cf334de P 8ff43444b87a45ec97617f5fe374dad4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 14:52:18.565950 19133 raft_consensus.cc:729] T 23028294819a44e2926efbf60cf334de P 8ff43444b87a45ec97617f5fe374dad4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8ff43444b87a45ec97617f5fe374dad4, State: Initialized, Role: FOLLOWER
I20260620 14:52:18.566833 19133 consensus_queue.cc:260] T 23028294819a44e2926efbf60cf334de P 8ff43444b87a45ec97617f5fe374dad4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "07a75a6c306d4a03acc972cccb741e6e" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 39011 } } peers { permanent_uuid: "8ff43444b87a45ec97617f5fe374dad4" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 42185 } } peers { permanent_uuid: "cfcfe512d7a1477b87ee3370a4b487b5" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 46621 } }
I20260620 14:52:18.568382 19114 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:52:18.559977 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.2:42185 (local address 127.0.97.3:51283)
0620 14:52:18.560186 (+ 209us) negotiation.cc:107] Waiting for socket to connect
0620 14:52:18.560207 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0620 14:52:18.560332 (+ 125us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 14:52:18.561230 (+ 898us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 14:52:18.561235 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 14:52:18.561261 (+ 26us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 14:52:18.561538 (+ 277us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:52:18.561551 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:52:18.562946 (+ 1395us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:52:18.562951 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:52:18.565056 (+ 2105us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:52:18.565074 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:52:18.567483 (+ 2409us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:52:18.567489 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:52:18.567754 (+ 265us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:52:18.567801 (+ 47us) client_negotiation.cc:769] Sending connection context
0620 14:52:18.567900 (+ 99us) client_negotiation.cc:240] Negotiation successful
0620 14:52:18.568055 (+ 155us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":45}
I20260620 14:52:18.568662 19127 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:52:18.560562 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.3:51283 (local address 127.0.97.2:42185)
0620 14:52:18.560758 (+ 196us) server_negotiation.cc:206] Beginning negotiation
0620 14:52:18.560764 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 14:52:18.560783 (+ 19us) server_negotiation.cc:407] Connection header received
0620 14:52:18.560851 (+ 68us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:52:18.560855 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:52:18.560927 (+ 72us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:52:18.561021 (+ 94us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 14:52:18.561736 (+ 715us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:52:18.562774 (+ 1038us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:52:18.565309 (+ 2535us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:52:18.567275 (+ 1966us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:52:18.567476 (+ 201us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:52:18.567660 (+ 184us) server_negotiation.cc:1035] Waiting for connection context
0620 14:52:18.568266 (+ 606us) server_negotiation.cc:299] Negotiation successful
0620 14:52:18.568411 (+ 145us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":80}
I20260620 14:52:18.569603 18933 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "23028294819a44e2926efbf60cf334de" candidate_uuid: "07a75a6c306d4a03acc972cccb741e6e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8ff43444b87a45ec97617f5fe374dad4" is_pre_election: true
I20260620 14:52:18.569991 19138 leader_election.cc:288] T 23028294819a44e2926efbf60cf334de P 07a75a6c306d4a03acc972cccb741e6e [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 8ff43444b87a45ec97617f5fe374dad4 (127.0.97.2:42185), cfcfe512d7a1477b87ee3370a4b487b5 (127.0.97.1:46621)
I20260620 14:52:18.570195 18933 raft_consensus.cc:2454] T 23028294819a44e2926efbf60cf334de P 8ff43444b87a45ec97617f5fe374dad4 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 07a75a6c306d4a03acc972cccb741e6e in term 0.
W20260620 14:52:18.570624 18845 tablet.cc:2369] T 23028294819a44e2926efbf60cf334de P cfcfe512d7a1477b87ee3370a4b487b5: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260620 14:52:18.571097 19001 leader_election.cc:302] T 23028294819a44e2926efbf60cf334de P 07a75a6c306d4a03acc972cccb741e6e [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 07a75a6c306d4a03acc972cccb741e6e, 8ff43444b87a45ec97617f5fe374dad4; no voters:
I20260620 14:52:18.571578 19138 raft_consensus.cc:2781] T 23028294819a44e2926efbf60cf334de P 07a75a6c306d4a03acc972cccb741e6e [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260620 14:52:18.571755 19138 raft_consensus.cc:482] T 23028294819a44e2926efbf60cf334de P 07a75a6c306d4a03acc972cccb741e6e [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260620 14:52:18.571858 19138 raft_consensus.cc:3037] T 23028294819a44e2926efbf60cf334de P 07a75a6c306d4a03acc972cccb741e6e [term 0 FOLLOWER]: Advancing to term 1
I20260620 14:52:18.571871 18978 heartbeater.cc:503] Master 127.0.97.62:38439 was elected leader, sending a full tablet report...
I20260620 14:52:18.575363 19138 raft_consensus.cc:504] T 23028294819a44e2926efbf60cf334de P 07a75a6c306d4a03acc972cccb741e6e [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "07a75a6c306d4a03acc972cccb741e6e" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 39011 } } peers { permanent_uuid: "8ff43444b87a45ec97617f5fe374dad4" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 42185 } } peers { permanent_uuid: "cfcfe512d7a1477b87ee3370a4b487b5" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 46621 } }
I20260620 14:52:18.576280 19138 leader_election.cc:288] T 23028294819a44e2926efbf60cf334de P 07a75a6c306d4a03acc972cccb741e6e [CANDIDATE]: Term 1 election: Requested vote from peers 8ff43444b87a45ec97617f5fe374dad4 (127.0.97.2:42185), cfcfe512d7a1477b87ee3370a4b487b5 (127.0.97.1:46621)
I20260620 14:52:18.576701 18933 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "23028294819a44e2926efbf60cf334de" candidate_uuid: "07a75a6c306d4a03acc972cccb741e6e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8ff43444b87a45ec97617f5fe374dad4"
I20260620 14:52:18.576933 18933 raft_consensus.cc:3037] T 23028294819a44e2926efbf60cf334de P 8ff43444b87a45ec97617f5fe374dad4 [term 0 FOLLOWER]: Advancing to term 1
I20260620 14:52:18.578130 19130 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:52:18.569684 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.3:41305 (local address 127.0.97.1:46621)
0620 14:52:18.569888 (+ 204us) server_negotiation.cc:206] Beginning negotiation
0620 14:52:18.569893 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 14:52:18.571074 (+ 1181us) server_negotiation.cc:407] Connection header received
0620 14:52:18.571153 (+ 79us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:52:18.571158 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:52:18.571237 (+ 79us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:52:18.571323 (+ 86us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 14:52:18.571964 (+ 641us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:52:18.573048 (+ 1084us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:52:18.575410 (+ 2362us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:52:18.576679 (+ 1269us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:52:18.576825 (+ 146us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:52:18.576936 (+ 111us) server_negotiation.cc:1035] Waiting for connection context
0620 14:52:18.577722 (+ 786us) server_negotiation.cc:299] Negotiation successful
0620 14:52:18.577866 (+ 144us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":64}
I20260620 14:52:18.578518 19114 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:52:18.569326 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:46621 (local address 127.0.97.3:41305)
0620 14:52:18.570815 (+ 1489us) negotiation.cc:107] Waiting for socket to connect
0620 14:52:18.570836 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0620 14:52:18.570959 (+ 123us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 14:52:18.571477 (+ 518us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 14:52:18.571484 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 14:52:18.571513 (+ 29us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 14:52:18.571767 (+ 254us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:52:18.571782 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:52:18.573211 (+ 1429us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:52:18.573217 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:52:18.575195 (+ 1978us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:52:18.575214 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:52:18.577165 (+ 1951us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:52:18.577178 (+ 13us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:52:18.577506 (+ 328us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:52:18.577566 (+ 60us) client_negotiation.cc:769] Sending connection context
0620 14:52:18.577695 (+ 129us) client_negotiation.cc:240] Negotiation successful
0620 14:52:18.577907 (+ 212us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1372}
I20260620 14:52:18.579313 19133 ts_tablet_manager.cc:1428] T 23028294819a44e2926efbf60cf334de P 8ff43444b87a45ec97617f5fe374dad4: Time spent starting tablet: real 0.035s user 0.015s sys 0.000s
I20260620 14:52:18.580225 19131 ts_tablet_manager.cc:1428] T 23028294819a44e2926efbf60cf334de P 07a75a6c306d4a03acc972cccb741e6e: Time spent starting tablet: real 0.041s user 0.018s sys 0.024s
I20260620 14:52:18.580382 18933 raft_consensus.cc:2454] T 23028294819a44e2926efbf60cf334de P 8ff43444b87a45ec97617f5fe374dad4 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 07a75a6c306d4a03acc972cccb741e6e in term 1.
I20260620 14:52:18.580874 19001 leader_election.cc:302] T 23028294819a44e2926efbf60cf334de P 07a75a6c306d4a03acc972cccb741e6e [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 07a75a6c306d4a03acc972cccb741e6e, 8ff43444b87a45ec97617f5fe374dad4; no voters:
I20260620 14:52:18.580754 18798 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "23028294819a44e2926efbf60cf334de" candidate_uuid: "07a75a6c306d4a03acc972cccb741e6e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "cfcfe512d7a1477b87ee3370a4b487b5"
I20260620 14:52:18.581233 18798 raft_consensus.cc:3037] T 23028294819a44e2926efbf60cf334de P cfcfe512d7a1477b87ee3370a4b487b5 [term 0 FOLLOWER]: Advancing to term 1
I20260620 14:52:18.581450 19137 raft_consensus.cc:2781] T 23028294819a44e2926efbf60cf334de P 07a75a6c306d4a03acc972cccb741e6e [term 1 FOLLOWER]: Leader election won for term 1
I20260620 14:52:18.581646 19137 raft_consensus.cc:686] T 23028294819a44e2926efbf60cf334de P 07a75a6c306d4a03acc972cccb741e6e [term 1 LEADER]: Becoming Leader. State: Replica: 07a75a6c306d4a03acc972cccb741e6e, State: Running, Role: LEADER
I20260620 14:52:18.582230 19137 consensus_queue.cc:237] T 23028294819a44e2926efbf60cf334de P 07a75a6c306d4a03acc972cccb741e6e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "07a75a6c306d4a03acc972cccb741e6e" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 39011 } } peers { permanent_uuid: "8ff43444b87a45ec97617f5fe374dad4" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 42185 } } peers { permanent_uuid: "cfcfe512d7a1477b87ee3370a4b487b5" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 46621 } }
I20260620 14:52:18.581229 18799 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "23028294819a44e2926efbf60cf334de" candidate_uuid: "07a75a6c306d4a03acc972cccb741e6e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "cfcfe512d7a1477b87ee3370a4b487b5" is_pre_election: true
I20260620 14:52:18.584861 18798 raft_consensus.cc:2454] T 23028294819a44e2926efbf60cf334de P cfcfe512d7a1477b87ee3370a4b487b5 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 07a75a6c306d4a03acc972cccb741e6e in term 1.
I20260620 14:52:18.587457 18655 catalog_manager.cc:5495] T 23028294819a44e2926efbf60cf334de P 07a75a6c306d4a03acc972cccb741e6e reported cstate change: term changed from 0 to 1, leader changed from <none> to 07a75a6c306d4a03acc972cccb741e6e (127.0.97.3). New cstate: current_term: 1 leader_uuid: "07a75a6c306d4a03acc972cccb741e6e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "07a75a6c306d4a03acc972cccb741e6e" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 39011 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "8ff43444b87a45ec97617f5fe374dad4" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 42185 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "cfcfe512d7a1477b87ee3370a4b487b5" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 46621 } health_report { overall_health: UNKNOWN } } }
W20260620 14:52:18.615128 18979 tablet.cc:2369] T 23028294819a44e2926efbf60cf334de P 8ff43444b87a45ec97617f5fe374dad4: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260620 14:52:18.628923 19128 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:52:18.623174 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55770 (local address 127.0.97.3:39011)
0620 14:52:18.623365 (+ 191us) server_negotiation.cc:206] Beginning negotiation
0620 14:52:18.623371 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 14:52:18.623668 (+ 297us) server_negotiation.cc:407] Connection header received
0620 14:52:18.623873 (+ 205us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:52:18.623877 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:52:18.623962 (+ 85us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:52:18.624099 (+ 137us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 14:52:18.624458 (+ 359us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:52:18.625174 (+ 716us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:52:18.626589 (+ 1415us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:52:18.626969 (+ 380us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:52:18.627059 (+ 90us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:52:18.627714 (+ 655us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 14:52:18.628273 (+ 559us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0620 14:52:18.628403 (+ 130us) server_negotiation.cc:1035] Waiting for connection context
0620 14:52:18.628543 (+ 140us) server_negotiation.cc:299] Negotiation successful
0620 14:52:18.628585 (+ 42us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":55}
I20260620 14:52:18.637815 18798 raft_consensus.cc:1261] T 23028294819a44e2926efbf60cf334de P cfcfe512d7a1477b87ee3370a4b487b5 [term 1 FOLLOWER]: Refusing update from remote peer 07a75a6c306d4a03acc972cccb741e6e: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 14:52:18.637923 18933 raft_consensus.cc:1261] T 23028294819a44e2926efbf60cf334de P 8ff43444b87a45ec97617f5fe374dad4 [term 1 FOLLOWER]: Refusing update from remote peer 07a75a6c306d4a03acc972cccb741e6e: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 14:52:18.638909 19137 consensus_queue.cc:1035] T 23028294819a44e2926efbf60cf334de P 07a75a6c306d4a03acc972cccb741e6e [LEADER]: Connected to new peer: Peer: permanent_uuid: "cfcfe512d7a1477b87ee3370a4b487b5" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 46621 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 14:52:18.639284 19138 consensus_queue.cc:1035] T 23028294819a44e2926efbf60cf334de P 07a75a6c306d4a03acc972cccb741e6e [LEADER]: Connected to new peer: Peer: permanent_uuid: "8ff43444b87a45ec97617f5fe374dad4" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 42185 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 14:52:18.672933 18655 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:44524:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 14:52:18.673233 18655 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:44524:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 14:52:18.677062 18655 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 92fc2fdb55334e25980592943ab9f8f2: Sending DeleteTablet for 3 replicas of tablet 23028294819a44e2926efbf60cf334de
I20260620 14:52:18.678375 18913 tablet_service.cc:1514] Processing DeleteTablet for tablet 23028294819a44e2926efbf60cf334de with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 14:52:18 UTC) from {username='kudu', principal='kudu/127.0.97.62@KRBTEST.COM'} at 127.0.0.1:34114
I20260620 14:52:18.678375 18779 tablet_service.cc:1514] Processing DeleteTablet for tablet 23028294819a44e2926efbf60cf334de with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 14:52:18 UTC) from {username='kudu', principal='kudu/127.0.97.62@KRBTEST.COM'} at 127.0.0.1:55892
I20260620 14:52:18.678385 19047 tablet_service.cc:1514] Processing DeleteTablet for tablet 23028294819a44e2926efbf60cf334de with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 14:52:18 UTC) from {username='kudu', principal='kudu/127.0.97.62@KRBTEST.COM'} at 127.0.0.1:55768
I20260620 14:52:18.683266 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskUci3DU/build/asan/bin/kudu pid 18716
I20260620 14:52:18.690958 19152 tablet_replica.cc:330] T 23028294819a44e2926efbf60cf334de P cfcfe512d7a1477b87ee3370a4b487b5: stopping tablet replica
I20260620 14:52:18.691582 19152 raft_consensus.cc:2229] T 23028294819a44e2926efbf60cf334de P cfcfe512d7a1477b87ee3370a4b487b5 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 14:52:18.692332 19152 raft_consensus.cc:2258] T 23028294819a44e2926efbf60cf334de P cfcfe512d7a1477b87ee3370a4b487b5 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 14:52:18.694931 19152 ts_tablet_manager.cc:1905] T 23028294819a44e2926efbf60cf334de P cfcfe512d7a1477b87ee3370a4b487b5: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 14:52:18.696695 19153 tablet_replica.cc:330] T 23028294819a44e2926efbf60cf334de P 8ff43444b87a45ec97617f5fe374dad4: stopping tablet replica
I20260620 14:52:18.697232 19153 raft_consensus.cc:2229] T 23028294819a44e2926efbf60cf334de P 8ff43444b87a45ec97617f5fe374dad4 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 14:52:18.697880 19154 tablet_replica.cc:330] T 23028294819a44e2926efbf60cf334de P 07a75a6c306d4a03acc972cccb741e6e: stopping tablet replica
I20260620 14:52:18.697877 19153 raft_consensus.cc:2258] T 23028294819a44e2926efbf60cf334de P 8ff43444b87a45ec97617f5fe374dad4 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 14:52:18.698393 19154 raft_consensus.cc:2229] T 23028294819a44e2926efbf60cf334de P 07a75a6c306d4a03acc972cccb741e6e [term 1 LEADER]: Raft consensus shutting down.
I20260620 14:52:18.700467 19153 ts_tablet_manager.cc:1905] T 23028294819a44e2926efbf60cf334de P 8ff43444b87a45ec97617f5fe374dad4: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 14:52:18.701049 19154 raft_consensus.cc:2258] T 23028294819a44e2926efbf60cf334de P 07a75a6c306d4a03acc972cccb741e6e [term 1 FOLLOWER]: Raft consensus is shut down!
Jun 20 14:52:18 dist-test-slave-q9j6 krb5kdc[18610](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781967136, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.1@KRBTEST.COM
I20260620 14:52:18.705771 19152 ts_tablet_manager.cc:1918] T 23028294819a44e2926efbf60cf334de P cfcfe512d7a1477b87ee3370a4b487b5: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260620 14:52:18.706115 19152 log.cc:1196] T 23028294819a44e2926efbf60cf334de P cfcfe512d7a1477b87ee3370a4b487b5: Deleting WAL directory at /tmp/dist-test-taskUci3DU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781966829091300-388-0/minicluster-data/ts-0/wal/wals/23028294819a44e2926efbf60cf334de
I20260620 14:52:18.706749 19152 ts_tablet_manager.cc:1939] T 23028294819a44e2926efbf60cf334de P cfcfe512d7a1477b87ee3370a4b487b5: Deleting consensus metadata
I20260620 14:52:18.707799 19154 ts_tablet_manager.cc:1905] T 23028294819a44e2926efbf60cf334de P 07a75a6c306d4a03acc972cccb741e6e: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 14:52:18.709111 19153 ts_tablet_manager.cc:1918] T 23028294819a44e2926efbf60cf334de P 8ff43444b87a45ec97617f5fe374dad4: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260620 14:52:18.709075 18639 catalog_manager.cc:4841] TS cfcfe512d7a1477b87ee3370a4b487b5 (127.0.97.1:46621): tablet 23028294819a44e2926efbf60cf334de (table test-table [id=41b053796bd14d5ab5db998935c1f70b]) successfully deleted
I20260620 14:52:18.709299 19153 log.cc:1196] T 23028294819a44e2926efbf60cf334de P 8ff43444b87a45ec97617f5fe374dad4: Deleting WAL directory at /tmp/dist-test-taskUci3DU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781966829091300-388-0/minicluster-data/ts-1/wal/wals/23028294819a44e2926efbf60cf334de
I20260620 14:52:18.709829 19153 ts_tablet_manager.cc:1939] T 23028294819a44e2926efbf60cf334de P 8ff43444b87a45ec97617f5fe374dad4: Deleting consensus metadata
I20260620 14:52:18.711786 18640 catalog_manager.cc:4841] TS 8ff43444b87a45ec97617f5fe374dad4 (127.0.97.2:42185): tablet 23028294819a44e2926efbf60cf334de (table test-table [id=41b053796bd14d5ab5db998935c1f70b]) successfully deleted
I20260620 14:52:18.715233 19154 ts_tablet_manager.cc:1918] T 23028294819a44e2926efbf60cf334de P 07a75a6c306d4a03acc972cccb741e6e: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260620 14:52:18.715476 19154 log.cc:1196] T 23028294819a44e2926efbf60cf334de P 07a75a6c306d4a03acc972cccb741e6e: Deleting WAL directory at /tmp/dist-test-taskUci3DU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781966829091300-388-0/minicluster-data/ts-2/wal/wals/23028294819a44e2926efbf60cf334de
I20260620 14:52:18.716140 19154 ts_tablet_manager.cc:1939] T 23028294819a44e2926efbf60cf334de P 07a75a6c306d4a03acc972cccb741e6e: Deleting consensus metadata
I20260620 14:52:18.717868 19130 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:52:18.684787 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55906 (local address 127.0.97.1:46621)
0620 14:52:18.684988 (+ 201us) server_negotiation.cc:206] Beginning negotiation
0620 14:52:18.684996 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0620 14:52:18.687544 (+ 2548us) server_negotiation.cc:407] Connection header received
0620 14:52:18.688528 (+ 984us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:52:18.688538 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:52:18.688700 (+ 162us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:52:18.688796 (+ 96us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:52:18.693211 (+ 4415us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:52:18.694382 (+ 1171us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:52:18.698514 (+ 4132us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:52:18.699241 (+ 727us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:52:18.699409 (+ 168us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:52:18.709833 (+ 10424us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 14:52:18.709850 (+ 17us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 14:52:18.709872 (+ 22us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 14:52:18.709942 (+ 70us) server_negotiation.cc:922] calling sasl_server_start()
0620 14:52:18.714224 (+ 4282us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:52:18.715342 (+ 1118us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:52:18.715355 (+ 13us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:52:18.715360 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:52:18.715561 (+ 201us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:52:18.716081 (+ 520us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:52:18.716087 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:52:18.716090 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:52:18.716570 (+ 480us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 14:52:18.716886 (+ 316us) server_negotiation.cc:1035] Waiting for connection context
0620 14:52:18.717223 (+ 337us) server_negotiation.cc:299] Negotiation successful
0620 14:52:18.717472 (+ 249us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":66}
I20260620 14:52:18.718082 18642 catalog_manager.cc:4841] TS 07a75a6c306d4a03acc972cccb741e6e (127.0.97.3:39011): tablet 23028294819a44e2926efbf60cf334de (table test-table [id=41b053796bd14d5ab5db998935c1f70b]) successfully deleted
I20260620 14:52:18.718629 18839 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 14:52:18.812496 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskUci3DU/build/asan/bin/kudu with pid 18716
I20260620 14:52:18.830909 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskUci3DU/build/asan/bin/kudu pid 18850
Jun 20 14:52:18 dist-test-slave-q9j6 krb5kdc[18610](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781967136, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.2@KRBTEST.COM
I20260620 14:52:18.848701 19127 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:52:18.831753 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34122 (local address 127.0.97.2:42185)
0620 14:52:18.831929 (+ 176us) server_negotiation.cc:206] Beginning negotiation
0620 14:52:18.831934 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 14:52:18.832083 (+ 149us) server_negotiation.cc:407] Connection header received
0620 14:52:18.832255 (+ 172us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:52:18.832259 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:52:18.832324 (+ 65us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:52:18.832412 (+ 88us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:52:18.834120 (+ 1708us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:52:18.834842 (+ 722us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:52:18.836730 (+ 1888us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:52:18.837120 (+ 390us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:52:18.837215 (+ 95us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:52:18.842634 (+ 5419us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 14:52:18.842651 (+ 17us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 14:52:18.842667 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 14:52:18.842713 (+ 46us) server_negotiation.cc:922] calling sasl_server_start()
0620 14:52:18.845709 (+ 2996us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:52:18.846486 (+ 777us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:52:18.846494 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:52:18.846497 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:52:18.846633 (+ 136us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:52:18.847034 (+ 401us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:52:18.847038 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:52:18.847040 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:52:18.847432 (+ 392us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 14:52:18.847759 (+ 327us) server_negotiation.cc:1035] Waiting for connection context
0620 14:52:18.848120 (+ 361us) server_negotiation.cc:299] Negotiation successful
0620 14:52:18.848359 (+ 239us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":71}
I20260620 14:52:18.849359 18973 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 14:52:18.944680 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskUci3DU/build/asan/bin/kudu with pid 18850
I20260620 14:52:18.964340 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskUci3DU/build/asan/bin/kudu pid 18984
Jun 20 14:52:18 dist-test-slave-q9j6 krb5kdc[18610](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781967136, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.3@KRBTEST.COM
I20260620 14:52:18.983439 19128 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:52:18.965185 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55778 (local address 127.0.97.3:39011)
0620 14:52:18.965372 (+ 187us) server_negotiation.cc:206] Beginning negotiation
0620 14:52:18.965377 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 14:52:18.965425 (+ 48us) server_negotiation.cc:407] Connection header received
0620 14:52:18.965648 (+ 223us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:52:18.965651 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:52:18.965716 (+ 65us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:52:18.965798 (+ 82us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:52:18.967739 (+ 1941us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:52:18.968531 (+ 792us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:52:18.970433 (+ 1902us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:52:18.971155 (+ 722us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:52:18.971312 (+ 157us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:52:18.975839 (+ 4527us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 14:52:18.975863 (+ 24us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 14:52:18.975884 (+ 21us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 14:52:18.975952 (+ 68us) server_negotiation.cc:922] calling sasl_server_start()
0620 14:52:18.980323 (+ 4371us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:52:18.981136 (+ 813us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:52:18.981146 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:52:18.981149 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:52:18.981334 (+ 185us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:52:18.981736 (+ 402us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:52:18.981742 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:52:18.981745 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:52:18.982263 (+ 518us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 14:52:18.982595 (+ 332us) server_negotiation.cc:1035] Waiting for connection context
0620 14:52:18.982770 (+ 175us) server_negotiation.cc:299] Negotiation successful
0620 14:52:18.983020 (+ 250us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":87}
I20260620 14:52:18.984294 19107 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 14:52:19.110291 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskUci3DU/build/asan/bin/kudu with pid 18984
I20260620 14:52:19.130692 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskUci3DU/build/asan/bin/kudu pid 18623
I20260620 14:52:19.131913 18685 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 14:52:19.245497 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskUci3DU/build/asan/bin/kudu with pid 18623
2026-06-20T14:52:19Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/2 (4456 ms)
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/3
2026-06-20T14:52:19Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T14:52:19Z Disabled control of system clock
I20260620 14:52:19.325141 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskUci3DU/build/asan/bin/kudu
/tmp/dist-test-taskUci3DU/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskUci3DU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781966829091300-388-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskUci3DU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781966829091300-388-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskUci3DU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781966829091300-388-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskUci3DU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781966829091300-388-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:32963
--webserver_interface=127.0.97.62
--webserver_port=0
--builtin_ntp_servers=127.0.97.20:34929
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.97.62:32963
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260620 14:52:19.667829 19163 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 14:52:19.668154 19163 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 14:52:19.668243 19163 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 14:52:19.675697 19163 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 14:52:19.675820 19163 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 14:52:19.675856 19163 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 14:52:19.675908 19163 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 14:52:19.675953 19163 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 14:52:19.685397 19163 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:34929
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskUci3DU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781966829091300-388-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskUci3DU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781966829091300-388-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.97.62:32963
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:32963
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskUci3DU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781966829091300-388-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.97.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskUci3DU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781966829091300-388-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 13:41:20 UTC on ccycloud.quasar-vdagoi.root.comops.site
build id 4457462
ASAN enabled
I20260620 14:52:19.686674 19163 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 14:52:19.688454 19163 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 14:52:19.700871 19171 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 14:52:19.700953 19169 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 14:52:19.700901 19168 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 14:52:19.702193 19163 server_base.cc:1034] running on GCE node
I20260620 14:52:19.703094 19163 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 14:52:19.704701 19163 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 14:52:19.705952 19163 hybrid_clock.cc:648] HybridClock initialized: now 1781967139705917 us; error 55 us; skew 500 ppm
I20260620 14:52:19.706360 19163 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260620 14:52:19.708747 19163 webserver.cc:460] Webserver started at http://127.0.97.62:45099/ using document root <none> and password file <none>
I20260620 14:52:19.709548 19163 fs_manager.cc:359] Metadata directory not provided
I20260620 14:52:19.709679 19163 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 14:52:19.710016 19163 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 14:52:19.713074 19163 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskUci3DU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781966829091300-388-0/minicluster-data/master-0/data/instance:
uuid: "809bcdc911834f1a98bf8e28ad2a41c4"
format_stamp: "Formatted at 2026-06-20 14:52:19 on dist-test-slave-q9j6"
server_key: "5c3f1383e8a038355304b9bc48d83e84"
server_key_iv: "fcbfa4c63e9801048921f948eba3b789"
server_key_version: "encryptionkey@0"
I20260620 14:52:19.713887 19163 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskUci3DU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781966829091300-388-0/minicluster-data/master-0/wal/instance:
uuid: "809bcdc911834f1a98bf8e28ad2a41c4"
format_stamp: "Formatted at 2026-06-20 14:52:19 on dist-test-slave-q9j6"
server_key: "5c3f1383e8a038355304b9bc48d83e84"
server_key_iv: "fcbfa4c63e9801048921f948eba3b789"
server_key_version: "encryptionkey@0"
I20260620 14:52:19.719219 19163 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.000s
I20260620 14:52:19.722863 19177 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 14:52:19.724399 19163 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260620 14:52:19.724596 19163 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskUci3DU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781966829091300-388-0/minicluster-data/master-0/data,/tmp/dist-test-taskUci3DU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781966829091300-388-0/minicluster-data/master-0/wal
uuid: "809bcdc911834f1a98bf8e28ad2a41c4"
format_stamp: "Formatted at 2026-06-20 14:52:19 on dist-test-slave-q9j6"
server_key: "5c3f1383e8a038355304b9bc48d83e84"
server_key_iv: "fcbfa4c63e9801048921f948eba3b789"
server_key_version: "encryptionkey@0"
I20260620 14:52:19.724829 19163 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskUci3DU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781966829091300-388-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskUci3DU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781966829091300-388-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskUci3DU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781966829091300-388-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 14:52:19.746095 19163 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 14:52:19.747004 19163 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 14:52:19.747294 19163 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 14:52:19.766260 19163 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.62:32963
I20260620 14:52:19.766285 19228 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.62:32963 every 8 connection(s)
I20260620 14:52:19.767694 19163 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskUci3DU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781966829091300-388-0/minicluster-data/master-0/data/info.pb
I20260620 14:52:19.770656 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskUci3DU/build/asan/bin/kudu as pid 19163
I20260620 14:52:19.770830 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskUci3DU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781966829091300-388-0/minicluster-data/master-0/wal/instance
I20260620 14:52:19.771263 388 external_mini_cluster.cc:1428] Setting key 761539a9c28a121f792e939662f214ae
I20260620 14:52:19.771713 19229 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 14:52:19.785382 19229 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 809bcdc911834f1a98bf8e28ad2a41c4: Bootstrap starting.
I20260620 14:52:19.786026 19232 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:52:19.773118 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35964 (local address 127.0.97.62:32963)
0620 14:52:19.774107 (+ 989us) server_negotiation.cc:206] Beginning negotiation
0620 14:52:19.774119 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0620 14:52:19.774204 (+ 85us) server_negotiation.cc:407] Connection header received
0620 14:52:19.776898 (+ 2694us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:52:19.776926 (+ 28us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:52:19.777427 (+ 501us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:52:19.777858 (+ 431us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:52:19.778477 (+ 619us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:52:19.779815 (+ 1338us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:52:19.781219 (+ 1404us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:52:19.781992 (+ 773us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:52:19.782214 (+ 222us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:52:19.783471 (+ 1257us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 14:52:19.783479 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 14:52:19.783496 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 14:52:19.784157 (+ 661us) server_negotiation.cc:922] calling sasl_server_start()
0620 14:52:19.784238 (+ 81us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 14:52:19.784259 (+ 21us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 14:52:19.784390 (+ 131us) server_negotiation.cc:1035] Waiting for connection context
0620 14:52:19.784632 (+ 242us) server_negotiation.cc:299] Negotiation successful
0620 14:52:19.784971 (+ 339us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":570,"thread_start_us":370,"threads_started":1}
I20260620 14:52:19.789352 19229 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 809bcdc911834f1a98bf8e28ad2a41c4: Neither blocks nor log segments found. Creating new log.
I20260620 14:52:19.790369 19229 log.cc:824] T 00000000000000000000000000000000 P 809bcdc911834f1a98bf8e28ad2a41c4: Log is configured to *not* fsync() on all Append() calls
I20260620 14:52:19.793497 19229 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 809bcdc911834f1a98bf8e28ad2a41c4: No bootstrap required, opened a new log
I20260620 14:52:19.801273 19229 raft_consensus.cc:348] T 00000000000000000000000000000000 P 809bcdc911834f1a98bf8e28ad2a41c4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "809bcdc911834f1a98bf8e28ad2a41c4" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 32963 } }
I20260620 14:52:19.801890 19229 raft_consensus.cc:374] T 00000000000000000000000000000000 P 809bcdc911834f1a98bf8e28ad2a41c4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 14:52:19.802008 19229 raft_consensus.cc:729] T 00000000000000000000000000000000 P 809bcdc911834f1a98bf8e28ad2a41c4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 809bcdc911834f1a98bf8e28ad2a41c4, State: Initialized, Role: FOLLOWER
I20260620 14:52:19.802661 19229 consensus_queue.cc:260] T 00000000000000000000000000000000 P 809bcdc911834f1a98bf8e28ad2a41c4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "809bcdc911834f1a98bf8e28ad2a41c4" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 32963 } }
I20260620 14:52:19.802872 19229 raft_consensus.cc:388] T 00000000000000000000000000000000 P 809bcdc911834f1a98bf8e28ad2a41c4 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 14:52:19.802974 19229 raft_consensus.cc:482] T 00000000000000000000000000000000 P 809bcdc911834f1a98bf8e28ad2a41c4 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 14:52:19.803099 19229 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 809bcdc911834f1a98bf8e28ad2a41c4 [term 0 FOLLOWER]: Advancing to term 1
I20260620 14:52:19.805362 19229 raft_consensus.cc:504] T 00000000000000000000000000000000 P 809bcdc911834f1a98bf8e28ad2a41c4 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "809bcdc911834f1a98bf8e28ad2a41c4" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 32963 } }
I20260620 14:52:19.805858 19229 leader_election.cc:302] T 00000000000000000000000000000000 P 809bcdc911834f1a98bf8e28ad2a41c4 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 809bcdc911834f1a98bf8e28ad2a41c4; no voters:
I20260620 14:52:19.806406 19229 leader_election.cc:288] T 00000000000000000000000000000000 P 809bcdc911834f1a98bf8e28ad2a41c4 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260620 14:52:19.806607 19234 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 809bcdc911834f1a98bf8e28ad2a41c4 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 14:52:19.807296 19234 raft_consensus.cc:686] T 00000000000000000000000000000000 P 809bcdc911834f1a98bf8e28ad2a41c4 [term 1 LEADER]: Becoming Leader. State: Replica: 809bcdc911834f1a98bf8e28ad2a41c4, State: Running, Role: LEADER
I20260620 14:52:19.807876 19234 consensus_queue.cc:237] T 00000000000000000000000000000000 P 809bcdc911834f1a98bf8e28ad2a41c4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "809bcdc911834f1a98bf8e28ad2a41c4" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 32963 } }
I20260620 14:52:19.808691 19229 sys_catalog.cc:564] T 00000000000000000000000000000000 P 809bcdc911834f1a98bf8e28ad2a41c4 [sys.catalog]: configured and running, proceeding with master startup.
I20260620 14:52:19.810931 19236 sys_catalog.cc:455] T 00000000000000000000000000000000 P 809bcdc911834f1a98bf8e28ad2a41c4 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 809bcdc911834f1a98bf8e28ad2a41c4. Latest consensus state: current_term: 1 leader_uuid: "809bcdc911834f1a98bf8e28ad2a41c4" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "809bcdc911834f1a98bf8e28ad2a41c4" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 32963 } } }
I20260620 14:52:19.811343 19236 sys_catalog.cc:458] T 00000000000000000000000000000000 P 809bcdc911834f1a98bf8e28ad2a41c4 [sys.catalog]: This master's current role is: LEADER
I20260620 14:52:19.810837 19235 sys_catalog.cc:455] T 00000000000000000000000000000000 P 809bcdc911834f1a98bf8e28ad2a41c4 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "809bcdc911834f1a98bf8e28ad2a41c4" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "809bcdc911834f1a98bf8e28ad2a41c4" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 32963 } } }
I20260620 14:52:19.811789 19235 sys_catalog.cc:458] T 00000000000000000000000000000000 P 809bcdc911834f1a98bf8e28ad2a41c4 [sys.catalog]: This master's current role is: LEADER
I20260620 14:52:19.814363 19241 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 14:52:19.819788 19241 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 14:52:19.830257 19241 catalog_manager.cc:1347] Generated new cluster ID: 530e4abcd9424cbea684ce2cbf1b4457
I20260620 14:52:19.830399 19241 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 14:52:19.856513 19241 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 14:52:19.858413 19241 catalog_manager.cc:1504] Loading token signing keys...
I20260620 14:52:19.871822 19241 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 809bcdc911834f1a98bf8e28ad2a41c4: Generated new TSK 0
I20260620 14:52:19.872781 19241 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260620 14:52:19.885802 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskUci3DU/build/asan/bin/kudu
/tmp/dist-test-taskUci3DU/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskUci3DU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781966829091300-388-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskUci3DU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781966829091300-388-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskUci3DU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781966829091300-388-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskUci3DU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781966829091300-388-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.1:0
--local_ip_for_outbound_sockets=127.0.97.1
--webserver_interface=127.0.97.1
--webserver_port=0
--tserver_master_addrs=127.0.97.62:32963
--builtin_ntp_servers=127.0.97.20:34929
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260620 14:52:20.252570 19253 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 14:52:20.252894 19253 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 14:52:20.253064 19253 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 14:52:20.260800 19253 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 14:52:20.260928 19253 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 14:52:20.261078 19253 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.1
I20260620 14:52:20.270869 19253 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:34929
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskUci3DU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781966829091300-388-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskUci3DU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781966829091300-388-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskUci3DU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781966829091300-388-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.97.1
--webserver_port=0
--tserver_master_addrs=127.0.97.62:32963
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.1
--log_dir=/tmp/dist-test-taskUci3DU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781966829091300-388-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 13:41:20 UTC on ccycloud.quasar-vdagoi.root.comops.site
build id 4457462
ASAN enabled
I20260620 14:52:20.272324 19253 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 14:52:20.274255 19253 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 14:52:20.288528 19258 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 14:52:20.288528 19259 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 14:52:20.288619 19261 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 14:52:20.289381 19253 server_base.cc:1034] running on GCE node
I20260620 14:52:20.290131 19253 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 14:52:20.291226 19253 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 14:52:20.292488 19253 hybrid_clock.cc:648] HybridClock initialized: now 1781967140292387 us; error 105 us; skew 500 ppm
I20260620 14:52:20.292877 19253 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260620 14:52:20.295346 19253 webserver.cc:460] Webserver started at http://127.0.97.1:37043/ using document root <none> and password file <none>
I20260620 14:52:20.296211 19253 fs_manager.cc:359] Metadata directory not provided
I20260620 14:52:20.296355 19253 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 14:52:20.296703 19253 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 14:52:20.299901 19253 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskUci3DU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781966829091300-388-0/minicluster-data/ts-0/data/instance:
uuid: "b5cb93cc116b4642bbaa4356a44d5e19"
format_stamp: "Formatted at 2026-06-20 14:52:20 on dist-test-slave-q9j6"
server_key: "e05e83bd8499bf51aa1b00e0c0ba2ab5"
server_key_iv: "77a9866e38434bba7cf51ec87826c550"
server_key_version: "encryptionkey@0"
I20260620 14:52:20.300805 19253 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskUci3DU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781966829091300-388-0/minicluster-data/ts-0/wal/instance:
uuid: "b5cb93cc116b4642bbaa4356a44d5e19"
format_stamp: "Formatted at 2026-06-20 14:52:20 on dist-test-slave-q9j6"
server_key: "e05e83bd8499bf51aa1b00e0c0ba2ab5"
server_key_iv: "77a9866e38434bba7cf51ec87826c550"
server_key_version: "encryptionkey@0"
I20260620 14:52:20.306653 19253 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.000s
I20260620 14:52:20.311045 19267 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 14:52:20.312794 19253 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.003s sys 0.000s
I20260620 14:52:20.313066 19253 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskUci3DU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781966829091300-388-0/minicluster-data/ts-0/data,/tmp/dist-test-taskUci3DU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781966829091300-388-0/minicluster-data/ts-0/wal
uuid: "b5cb93cc116b4642bbaa4356a44d5e19"
format_stamp: "Formatted at 2026-06-20 14:52:20 on dist-test-slave-q9j6"
server_key: "e05e83bd8499bf51aa1b00e0c0ba2ab5"
server_key_iv: "77a9866e38434bba7cf51ec87826c550"
server_key_version: "encryptionkey@0"
I20260620 14:52:20.313376 19253 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskUci3DU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781966829091300-388-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskUci3DU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781966829091300-388-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskUci3DU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781966829091300-388-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 14:52:20.325450 19253 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 14:52:20.326424 19253 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 14:52:20.326746 19253 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 14:52:20.327901 19253 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 14:52:20.329632 19253 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 14:52:20.329777 19253 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 14:52:20.329921 19253 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 14:52:20.330015 19253 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 14:52:20.365294 19253 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.1:43201
I20260620 14:52:20.365339 19379 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.1:43201 every 8 connection(s)
I20260620 14:52:20.366791 19253 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskUci3DU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781966829091300-388-0/minicluster-data/ts-0/data/info.pb
I20260620 14:52:20.367345 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskUci3DU/build/asan/bin/kudu as pid 19253
I20260620 14:52:20.367532 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskUci3DU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781966829091300-388-0/minicluster-data/ts-0/wal/instance
I20260620 14:52:20.368239 388 external_mini_cluster.cc:1428] Setting key ca74a997aeb3957b80312acaea90009f
I20260620 14:52:20.371521 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskUci3DU/build/asan/bin/kudu
/tmp/dist-test-taskUci3DU/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskUci3DU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781966829091300-388-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskUci3DU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781966829091300-388-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskUci3DU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781966829091300-388-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskUci3DU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781966829091300-388-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.2:0
--local_ip_for_outbound_sockets=127.0.97.2
--webserver_interface=127.0.97.2
--webserver_port=0
--tserver_master_addrs=127.0.97.62:32963
--builtin_ntp_servers=127.0.97.20:34929
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
I20260620 14:52:20.382714 19382 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:52:20.369781 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.1:36521 (local address 127.0.97.62:32963)
0620 14:52:20.370307 (+ 526us) server_negotiation.cc:206] Beginning negotiation
0620 14:52:20.370311 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 14:52:20.371957 (+ 1646us) server_negotiation.cc:407] Connection header received
0620 14:52:20.373824 (+ 1867us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:52:20.373831 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:52:20.373929 (+ 98us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:52:20.374061 (+ 132us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:52:20.375722 (+ 1661us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:52:20.376450 (+ 728us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:52:20.379254 (+ 2804us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:52:20.379846 (+ 592us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:52:20.380005 (+ 159us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:52:20.381634 (+ 1629us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 14:52:20.381644 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 14:52:20.381652 (+ 8us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 14:52:20.381682 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0620 14:52:20.381735 (+ 53us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 14:52:20.381754 (+ 19us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 14:52:20.381853 (+ 99us) server_negotiation.cc:1035] Waiting for connection context
0620 14:52:20.382079 (+ 226us) server_negotiation.cc:299] Negotiation successful
0620 14:52:20.382209 (+ 130us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":400,"thread_start_us":277,"threads_started":1}
I20260620 14:52:20.383740 19383 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:52:20.370577 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:32963 (local address 127.0.97.1:36521)
0620 14:52:20.371756 (+ 1179us) negotiation.cc:107] Waiting for socket to connect
0620 14:52:20.371807 (+ 51us) client_negotiation.cc:174] Beginning negotiation
0620 14:52:20.373508 (+ 1701us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 14:52:20.374375 (+ 867us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 14:52:20.374391 (+ 16us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 14:52:20.374552 (+ 161us) client_negotiation.cc:189] Negotiated authn=SASL
0620 14:52:20.375434 (+ 882us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:52:20.375468 (+ 34us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:52:20.376661 (+ 1193us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:52:20.376670 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:52:20.379031 (+ 2361us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:52:20.379057 (+ 26us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:52:20.380101 (+ 1044us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:52:20.380106 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:52:20.380343 (+ 237us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:52:20.381198 (+ 855us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0620 14:52:20.381201 (+ 3us) client_negotiation.cc:656] calling sasl_client_start()
0620 14:52:20.381252 (+ 51us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0620 14:52:20.381365 (+ 113us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 14:52:20.381860 (+ 495us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 14:52:20.381869 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 14:52:20.381875 (+ 6us) client_negotiation.cc:769] Sending connection context
0620 14:52:20.382040 (+ 165us) client_negotiation.cc:240] Negotiation successful
0620 14:52:20.382397 (+ 357us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":569,"thread_start_us":366,"threads_started":1}
I20260620 14:52:20.385638 19380 heartbeater.cc:347] Connected to a master server at 127.0.97.62:32963
I20260620 14:52:20.386050 19380 heartbeater.cc:464] Registering TS with master...
I20260620 14:52:20.386924 19380 heartbeater.cc:511] Master 127.0.97.62:32963 requested a full tablet report, sending...
I20260620 14:52:20.389295 19194 ts_manager.cc:194] Registered new tserver with Master: b5cb93cc116b4642bbaa4356a44d5e19 (127.0.97.1:43201)
I20260620 14:52:20.390890 19194 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.97.1:36521
W20260620 14:52:20.722834 19384 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 14:52:20.723153 19384 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 14:52:20.723304 19384 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 14:52:20.731077 19384 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 14:52:20.731195 19384 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 14:52:20.731377 19384 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.2
I20260620 14:52:20.740993 19384 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:34929
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskUci3DU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781966829091300-388-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskUci3DU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781966829091300-388-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskUci3DU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781966829091300-388-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.97.2
--webserver_port=0
--tserver_master_addrs=127.0.97.62:32963
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.2
--log_dir=/tmp/dist-test-taskUci3DU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781966829091300-388-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 13:41:20 UTC on ccycloud.quasar-vdagoi.root.comops.site
build id 4457462
ASAN enabled
I20260620 14:52:20.742398 19384 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 14:52:20.744299 19384 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 14:52:20.757987 19390 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 14:52:20.758062 19389 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 14:52:20.758091 19392 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 14:52:20.759078 19384 server_base.cc:1034] running on GCE node
I20260620 14:52:20.759929 19384 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 14:52:20.761044 19384 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 14:52:20.762305 19384 hybrid_clock.cc:648] HybridClock initialized: now 1781967140762268 us; error 53 us; skew 500 ppm
I20260620 14:52:20.762712 19384 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260620 14:52:20.765399 19384 webserver.cc:460] Webserver started at http://127.0.97.2:39975/ using document root <none> and password file <none>
I20260620 14:52:20.766263 19384 fs_manager.cc:359] Metadata directory not provided
I20260620 14:52:20.766414 19384 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 14:52:20.766778 19384 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 14:52:20.770119 19384 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskUci3DU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781966829091300-388-0/minicluster-data/ts-1/data/instance:
uuid: "f2ea51c4c5ac49bc8ab03373169895ae"
format_stamp: "Formatted at 2026-06-20 14:52:20 on dist-test-slave-q9j6"
server_key: "c8c2e220cdf440cc4566052ba9fc500d"
server_key_iv: "b16cbb03b6d5cc44272a0c18f2b6547c"
server_key_version: "encryptionkey@0"
I20260620 14:52:20.771085 19384 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskUci3DU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781966829091300-388-0/minicluster-data/ts-1/wal/instance:
uuid: "f2ea51c4c5ac49bc8ab03373169895ae"
format_stamp: "Formatted at 2026-06-20 14:52:20 on dist-test-slave-q9j6"
server_key: "c8c2e220cdf440cc4566052ba9fc500d"
server_key_iv: "b16cbb03b6d5cc44272a0c18f2b6547c"
server_key_version: "encryptionkey@0"
I20260620 14:52:20.777268 19384 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.003s sys 0.005s
I20260620 14:52:20.781627 19398 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 14:52:20.783435 19384 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.001s sys 0.000s
I20260620 14:52:20.783722 19384 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskUci3DU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781966829091300-388-0/minicluster-data/ts-1/data,/tmp/dist-test-taskUci3DU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781966829091300-388-0/minicluster-data/ts-1/wal
uuid: "f2ea51c4c5ac49bc8ab03373169895ae"
format_stamp: "Formatted at 2026-06-20 14:52:20 on dist-test-slave-q9j6"
server_key: "c8c2e220cdf440cc4566052ba9fc500d"
server_key_iv: "b16cbb03b6d5cc44272a0c18f2b6547c"
server_key_version: "encryptionkey@0"
I20260620 14:52:20.784017 19384 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskUci3DU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781966829091300-388-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskUci3DU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781966829091300-388-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskUci3DU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781966829091300-388-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 14:52:20.798722 19384 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 14:52:20.799727 19384 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 14:52:20.800021 19384 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 14:52:20.801077 19384 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 14:52:20.802754 19384 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 14:52:20.802886 19384 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 14:52:20.803027 19384 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 14:52:20.803115 19384 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 14:52:20.837957 19384 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.2:42301
I20260620 14:52:20.837994 19510 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.2:42301 every 8 connection(s)
I20260620 14:52:20.839440 19384 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskUci3DU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781966829091300-388-0/minicluster-data/ts-1/data/info.pb
I20260620 14:52:20.840761 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskUci3DU/build/asan/bin/kudu as pid 19384
I20260620 14:52:20.840941 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskUci3DU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781966829091300-388-0/minicluster-data/ts-1/wal/instance
I20260620 14:52:20.841539 388 external_mini_cluster.cc:1428] Setting key e2e8c80ae7de6ae66f4c2f0183d67a27
I20260620 14:52:20.844691 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskUci3DU/build/asan/bin/kudu
/tmp/dist-test-taskUci3DU/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskUci3DU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781966829091300-388-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskUci3DU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781966829091300-388-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskUci3DU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781966829091300-388-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskUci3DU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781966829091300-388-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.3:0
--local_ip_for_outbound_sockets=127.0.97.3
--webserver_interface=127.0.97.3
--webserver_port=0
--tserver_master_addrs=127.0.97.62:32963
--builtin_ntp_servers=127.0.97.20:34929
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
I20260620 14:52:20.855988 19382 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:52:20.842652 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.2:35837 (local address 127.0.97.62:32963)
0620 14:52:20.842831 (+ 179us) server_negotiation.cc:206] Beginning negotiation
0620 14:52:20.842835 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 14:52:20.844816 (+ 1981us) server_negotiation.cc:407] Connection header received
0620 14:52:20.846744 (+ 1928us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:52:20.846750 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:52:20.846841 (+ 91us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:52:20.846971 (+ 130us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:52:20.848371 (+ 1400us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:52:20.849587 (+ 1216us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:52:20.852578 (+ 2991us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:52:20.853101 (+ 523us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:52:20.853212 (+ 111us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:52:20.854981 (+ 1769us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 14:52:20.854986 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 14:52:20.854990 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 14:52:20.855010 (+ 20us) server_negotiation.cc:922] calling sasl_server_start()
0620 14:52:20.855043 (+ 33us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 14:52:20.855055 (+ 12us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 14:52:20.855126 (+ 71us) server_negotiation.cc:1035] Waiting for connection context
0620 14:52:20.855461 (+ 335us) server_negotiation.cc:299] Negotiation successful
0620 14:52:20.855551 (+ 90us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":59}
I20260620 14:52:20.857493 19513 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:52:20.843368 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:32963 (local address 127.0.97.2:35837)
0620 14:52:20.844642 (+ 1274us) negotiation.cc:107] Waiting for socket to connect
0620 14:52:20.844696 (+ 54us) client_negotiation.cc:174] Beginning negotiation
0620 14:52:20.846434 (+ 1738us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 14:52:20.847229 (+ 795us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 14:52:20.847239 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 14:52:20.847337 (+ 98us) client_negotiation.cc:189] Negotiated authn=SASL
0620 14:52:20.848157 (+ 820us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:52:20.848189 (+ 32us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:52:20.849776 (+ 1587us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:52:20.849784 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:52:20.852336 (+ 2552us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:52:20.852360 (+ 24us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:52:20.853357 (+ 997us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:52:20.853366 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:52:20.853724 (+ 358us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:52:20.854542 (+ 818us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0620 14:52:20.854548 (+ 6us) client_negotiation.cc:656] calling sasl_client_start()
0620 14:52:20.854608 (+ 60us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0620 14:52:20.854730 (+ 122us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 14:52:20.855188 (+ 458us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 14:52:20.855204 (+ 16us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 14:52:20.855216 (+ 12us) client_negotiation.cc:769] Sending connection context
0620 14:52:20.855444 (+ 228us) client_negotiation.cc:240] Negotiation successful
0620 14:52:20.855951 (+ 507us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":696,"thread_start_us":460,"threads_started":1}
I20260620 14:52:20.859302 19511 heartbeater.cc:347] Connected to a master server at 127.0.97.62:32963
I20260620 14:52:20.859702 19511 heartbeater.cc:464] Registering TS with master...
I20260620 14:52:20.860499 19511 heartbeater.cc:511] Master 127.0.97.62:32963 requested a full tablet report, sending...
I20260620 14:52:20.862223 19194 ts_manager.cc:194] Registered new tserver with Master: f2ea51c4c5ac49bc8ab03373169895ae (127.0.97.2:42301)
I20260620 14:52:20.863500 19194 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.97.2:35837
W20260620 14:52:21.197881 19514 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 14:52:21.198199 19514 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 14:52:21.198282 19514 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 14:52:21.206020 19514 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 14:52:21.206132 19514 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 14:52:21.206243 19514 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.3
I20260620 14:52:21.216133 19514 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:34929
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskUci3DU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781966829091300-388-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskUci3DU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781966829091300-388-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskUci3DU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781966829091300-388-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.97.3
--webserver_port=0
--tserver_master_addrs=127.0.97.62:32963
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.3
--log_dir=/tmp/dist-test-taskUci3DU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781966829091300-388-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 13:41:20 UTC on ccycloud.quasar-vdagoi.root.comops.site
build id 4457462
ASAN enabled
I20260620 14:52:21.217557 19514 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 14:52:21.219436 19514 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 14:52:21.234066 19520 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 14:52:21.234223 19522 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 14:52:21.234231 19519 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 14:52:21.235589 19514 server_base.cc:1034] running on GCE node
I20260620 14:52:21.236344 19514 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 14:52:21.237406 19514 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 14:52:21.238667 19514 hybrid_clock.cc:648] HybridClock initialized: now 1781967141238615 us; error 55 us; skew 500 ppm
I20260620 14:52:21.239037 19514 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260620 14:52:21.241551 19514 webserver.cc:460] Webserver started at http://127.0.97.3:34629/ using document root <none> and password file <none>
I20260620 14:52:21.242319 19514 fs_manager.cc:359] Metadata directory not provided
I20260620 14:52:21.242460 19514 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 14:52:21.242811 19514 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 14:52:21.246023 19514 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskUci3DU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781966829091300-388-0/minicluster-data/ts-2/data/instance:
uuid: "a048f73b63c14ea38806d99c64a644c1"
format_stamp: "Formatted at 2026-06-20 14:52:21 on dist-test-slave-q9j6"
server_key: "d1295946bb1e3aaf48fa4d9952c62d43"
server_key_iv: "68f77db53ac00c925b98867fdb2c037d"
server_key_version: "encryptionkey@0"
I20260620 14:52:21.247058 19514 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskUci3DU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781966829091300-388-0/minicluster-data/ts-2/wal/instance:
uuid: "a048f73b63c14ea38806d99c64a644c1"
format_stamp: "Formatted at 2026-06-20 14:52:21 on dist-test-slave-q9j6"
server_key: "d1295946bb1e3aaf48fa4d9952c62d43"
server_key_iv: "68f77db53ac00c925b98867fdb2c037d"
server_key_version: "encryptionkey@0"
I20260620 14:52:21.252990 19514 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.007s sys 0.001s
I20260620 14:52:21.257085 19528 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 14:52:21.258616 19514 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.001s
I20260620 14:52:21.258842 19514 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskUci3DU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781966829091300-388-0/minicluster-data/ts-2/data,/tmp/dist-test-taskUci3DU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781966829091300-388-0/minicluster-data/ts-2/wal
uuid: "a048f73b63c14ea38806d99c64a644c1"
format_stamp: "Formatted at 2026-06-20 14:52:21 on dist-test-slave-q9j6"
server_key: "d1295946bb1e3aaf48fa4d9952c62d43"
server_key_iv: "68f77db53ac00c925b98867fdb2c037d"
server_key_version: "encryptionkey@0"
I20260620 14:52:21.259115 19514 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskUci3DU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781966829091300-388-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskUci3DU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781966829091300-388-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskUci3DU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781966829091300-388-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 14:52:21.281013 19514 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 14:52:21.281975 19514 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 14:52:21.282277 19514 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 14:52:21.283365 19514 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 14:52:21.285101 19514 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 14:52:21.285233 19514 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 14:52:21.285374 19514 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 14:52:21.285465 19514 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 14:52:21.320072 19514 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.3:36173
I20260620 14:52:21.320161 19640 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.3:36173 every 8 connection(s)
I20260620 14:52:21.321458 19514 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskUci3DU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781966829091300-388-0/minicluster-data/ts-2/data/info.pb
I20260620 14:52:21.324432 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskUci3DU/build/asan/bin/kudu as pid 19514
I20260620 14:52:21.324605 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskUci3DU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781966829091300-388-0/minicluster-data/ts-2/wal/instance
I20260620 14:52:21.325006 388 external_mini_cluster.cc:1428] Setting key fb03736c9134108562d067b378ec0769
I20260620 14:52:21.334985 19382 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:52:21.324267 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.3:43363 (local address 127.0.97.62:32963)
0620 14:52:21.324509 (+ 242us) server_negotiation.cc:206] Beginning negotiation
0620 14:52:21.324517 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0620 14:52:21.326176 (+ 1659us) server_negotiation.cc:407] Connection header received
0620 14:52:21.328729 (+ 2553us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:52:21.328733 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:52:21.328807 (+ 74us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:52:21.328899 (+ 92us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:52:21.330045 (+ 1146us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:52:21.330809 (+ 764us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:52:21.332472 (+ 1663us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:52:21.332879 (+ 407us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:52:21.332976 (+ 97us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:52:21.334281 (+ 1305us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 14:52:21.334286 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 14:52:21.334289 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 14:52:21.334307 (+ 18us) server_negotiation.cc:922] calling sasl_server_start()
0620 14:52:21.334338 (+ 31us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 14:52:21.334349 (+ 11us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 14:52:21.334439 (+ 90us) server_negotiation.cc:1035] Waiting for connection context
0620 14:52:21.334637 (+ 198us) server_negotiation.cc:299] Negotiation successful
0620 14:52:21.334720 (+ 83us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":72}
I20260620 14:52:21.336107 19643 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:52:21.324827 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:32963 (local address 127.0.97.3:43363)
0620 14:52:21.325967 (+ 1140us) negotiation.cc:107] Waiting for socket to connect
0620 14:52:21.326043 (+ 76us) client_negotiation.cc:174] Beginning negotiation
0620 14:52:21.328465 (+ 2422us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 14:52:21.329116 (+ 651us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 14:52:21.329125 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 14:52:21.329222 (+ 97us) client_negotiation.cc:189] Negotiated authn=SASL
0620 14:52:21.329899 (+ 677us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:52:21.329918 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:52:21.330983 (+ 1065us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:52:21.330987 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:52:21.332330 (+ 1343us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:52:21.332343 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:52:21.333066 (+ 723us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:52:21.333071 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:52:21.333272 (+ 201us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:52:21.333956 (+ 684us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0620 14:52:21.333959 (+ 3us) client_negotiation.cc:656] calling sasl_client_start()
0620 14:52:21.334007 (+ 48us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0620 14:52:21.334088 (+ 81us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 14:52:21.334452 (+ 364us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 14:52:21.334459 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 14:52:21.334465 (+ 6us) client_negotiation.cc:769] Sending connection context
0620 14:52:21.334612 (+ 147us) client_negotiation.cc:240] Negotiation successful
0620 14:52:21.334915 (+ 303us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":554,"thread_start_us":334,"threads_started":1}
I20260620 14:52:21.337620 19641 heartbeater.cc:347] Connected to a master server at 127.0.97.62:32963
I20260620 14:52:21.337949 19641 heartbeater.cc:464] Registering TS with master...
I20260620 14:52:21.338801 19641 heartbeater.cc:511] Master 127.0.97.62:32963 requested a full tablet report, sending...
I20260620 14:52:21.340648 19193 ts_manager.cc:194] Registered new tserver with Master: a048f73b63c14ea38806d99c64a644c1 (127.0.97.3:36173)
I20260620 14:52:21.341542 19193 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.97.3:43363
I20260620 14:52:21.353228 388 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260620 14:52:21.363751 19382 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:52:21.357544 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35966 (local address 127.0.97.62:32963)
0620 14:52:21.357767 (+ 223us) server_negotiation.cc:206] Beginning negotiation
0620 14:52:21.357772 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 14:52:21.358178 (+ 406us) server_negotiation.cc:407] Connection header received
0620 14:52:21.358319 (+ 141us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:52:21.358323 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:52:21.358399 (+ 76us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:52:21.358492 (+ 93us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:52:21.359036 (+ 544us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:52:21.359838 (+ 802us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:52:21.361137 (+ 1299us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:52:21.361565 (+ 428us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:52:21.361666 (+ 101us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:52:21.362804 (+ 1138us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 14:52:21.362813 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 14:52:21.362819 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 14:52:21.362849 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0620 14:52:21.362884 (+ 35us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 14:52:21.362897 (+ 13us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 14:52:21.363020 (+ 123us) server_negotiation.cc:1035] Waiting for connection context
0620 14:52:21.363287 (+ 267us) server_negotiation.cc:299] Negotiation successful
0620 14:52:21.363371 (+ 84us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":65}
I20260620 14:52:21.368652 19193 catalog_manager.cc:1908] Servicing CreateTable request from {username='slave'} at 127.0.0.1:35966:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260620 14:52:21.376780 19193 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260620 14:52:21.394735 19380 heartbeater.cc:503] Master 127.0.97.62:32963 was elected leader, sending a full tablet report...
I20260620 14:52:21.414964 19655 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:52:21.390655 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:43201 (local address 127.0.0.1:37260)
0620 14:52:21.402915 (+ 12260us) negotiation.cc:107] Waiting for socket to connect
0620 14:52:21.402934 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0620 14:52:21.403160 (+ 226us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 14:52:21.404960 (+ 1800us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 14:52:21.404969 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 14:52:21.405005 (+ 36us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 14:52:21.405702 (+ 697us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:52:21.405720 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:52:21.408354 (+ 2634us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:52:21.408361 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:52:21.411297 (+ 2936us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:52:21.411314 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:52:21.413674 (+ 2360us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:52:21.413681 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:52:21.413942 (+ 261us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:52:21.414016 (+ 74us) client_negotiation.cc:769] Sending connection context
0620 14:52:21.414188 (+ 172us) client_negotiation.cc:240] Negotiation successful
0620 14:52:21.414452 (+ 264us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":12125,"mutex_wait_us":27,"spinlock_wait_cycles":11904,"thread_start_us":12700,"threads_started":1}
I20260620 14:52:21.415343 19656 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:52:21.390751 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38440 (local address 127.0.97.2:42301)
0620 14:52:21.406174 (+ 15423us) server_negotiation.cc:206] Beginning negotiation
0620 14:52:21.406184 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0620 14:52:21.406220 (+ 36us) server_negotiation.cc:407] Connection header received
0620 14:52:21.406346 (+ 126us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:52:21.406356 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:52:21.406667 (+ 311us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:52:21.406921 (+ 254us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 14:52:21.407935 (+ 1014us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:52:21.409553 (+ 1618us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:52:21.412493 (+ 2940us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:52:21.413761 (+ 1268us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:52:21.413905 (+ 144us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:52:21.414157 (+ 252us) server_negotiation.cc:1035] Waiting for connection context
0620 14:52:21.414572 (+ 415us) server_negotiation.cc:299] Negotiation successful
0620 14:52:21.414768 (+ 196us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":15244,"thread_start_us":15127,"threads_started":1}
I20260620 14:52:21.415367 19654 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:52:21.390670 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37260 (local address 127.0.97.1:43201)
0620 14:52:21.404148 (+ 13478us) server_negotiation.cc:206] Beginning negotiation
0620 14:52:21.404159 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0620 14:52:21.404195 (+ 36us) server_negotiation.cc:407] Connection header received
0620 14:52:21.404312 (+ 117us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:52:21.404337 (+ 25us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:52:21.404670 (+ 333us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:52:21.406003 (+ 1333us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 14:52:21.406305 (+ 302us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:52:21.408192 (+ 1887us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:52:21.411581 (+ 3389us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:52:21.413421 (+ 1840us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:52:21.413604 (+ 183us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:52:21.413985 (+ 381us) server_negotiation.cc:1035] Waiting for connection context
0620 14:52:21.414731 (+ 746us) server_negotiation.cc:299] Negotiation successful
0620 14:52:21.414959 (+ 228us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":13278,"thread_start_us":13148,"threads_started":1}
I20260620 14:52:21.416469 19658 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:52:21.406236 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.3:36173 (local address 127.0.0.1:39308)
0620 14:52:21.406829 (+ 593us) negotiation.cc:107] Waiting for socket to connect
0620 14:52:21.406847 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0620 14:52:21.407073 (+ 226us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 14:52:21.408091 (+ 1018us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 14:52:21.408095 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 14:52:21.408123 (+ 28us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 14:52:21.408550 (+ 427us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:52:21.408561 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:52:21.410592 (+ 2031us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:52:21.410598 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:52:21.413095 (+ 2497us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:52:21.413113 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:52:21.415532 (+ 2419us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:52:21.415541 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:52:21.415749 (+ 208us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:52:21.415787 (+ 38us) client_negotiation.cc:769] Sending connection context
0620 14:52:21.415965 (+ 178us) client_negotiation.cc:240] Negotiation successful
0620 14:52:21.416111 (+ 146us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":471,"mutex_wait_us":55,"thread_start_us":332,"threads_started":1}
I20260620 14:52:21.416671 19657 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:52:21.403921 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39308 (local address 127.0.97.3:36173)
0620 14:52:21.405696 (+ 1775us) server_negotiation.cc:206] Beginning negotiation
0620 14:52:21.405706 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0620 14:52:21.406958 (+ 1252us) server_negotiation.cc:407] Connection header received
0620 14:52:21.407269 (+ 311us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:52:21.407277 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:52:21.407555 (+ 278us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:52:21.407760 (+ 205us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 14:52:21.408785 (+ 1025us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:52:21.410395 (+ 1610us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:52:21.413347 (+ 2952us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:52:21.414955 (+ 1608us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:52:21.415096 (+ 141us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:52:21.415377 (+ 281us) server_negotiation.cc:1035] Waiting for connection context
0620 14:52:21.416017 (+ 640us) server_negotiation.cc:299] Negotiation successful
0620 14:52:21.416183 (+ 166us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1536,"thread_start_us":1394,"threads_started":1}
I20260620 14:52:21.417330 19653 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:52:21.390589 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.2:42301 (local address 127.0.0.1:38440)
0620 14:52:21.402844 (+ 12255us) negotiation.cc:107] Waiting for socket to connect
0620 14:52:21.402921 (+ 77us) client_negotiation.cc:174] Beginning negotiation
0620 14:52:21.403160 (+ 239us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 14:52:21.407212 (+ 4052us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 14:52:21.407218 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 14:52:21.407249 (+ 31us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 14:52:21.407716 (+ 467us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:52:21.407731 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:52:21.409729 (+ 1998us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:52:21.409736 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:52:21.412279 (+ 2543us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:52:21.412295 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:52:21.414152 (+ 1857us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:52:21.414162 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:52:21.414333 (+ 171us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:52:21.414375 (+ 42us) client_negotiation.cc:769] Sending connection context
0620 14:52:21.414459 (+ 84us) client_negotiation.cc:240] Negotiation successful
0620 14:52:21.414587 (+ 128us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":11809,"spinlock_wait_cycles":2560,"thread_start_us":11567,"threads_started":1}
I20260620 14:52:21.420053 19315 tablet_service.cc:1467] Processing CreateTablet for tablet 7b3ad3ddd4bb4376b0591449f3fae42f (DEFAULT_TABLE table=test-table [id=78ab1d797ff741d7a7f98ed94d0123bf]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 14:52:21.420702 19576 tablet_service.cc:1467] Processing CreateTablet for tablet 7b3ad3ddd4bb4376b0591449f3fae42f (DEFAULT_TABLE table=test-table [id=78ab1d797ff741d7a7f98ed94d0123bf]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 14:52:21.421618 19315 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 7b3ad3ddd4bb4376b0591449f3fae42f. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 14:52:21.422263 19576 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 7b3ad3ddd4bb4376b0591449f3fae42f. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 14:52:21.421963 19446 tablet_service.cc:1467] Processing CreateTablet for tablet 7b3ad3ddd4bb4376b0591449f3fae42f (DEFAULT_TABLE table=test-table [id=78ab1d797ff741d7a7f98ed94d0123bf]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 14:52:21.423381 19446 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 7b3ad3ddd4bb4376b0591449f3fae42f. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 14:52:21.436583 19659 tablet_bootstrap.cc:492] T 7b3ad3ddd4bb4376b0591449f3fae42f P b5cb93cc116b4642bbaa4356a44d5e19: Bootstrap starting.
I20260620 14:52:21.440488 19661 tablet_bootstrap.cc:492] T 7b3ad3ddd4bb4376b0591449f3fae42f P f2ea51c4c5ac49bc8ab03373169895ae: Bootstrap starting.
I20260620 14:52:21.440613 19659 tablet_bootstrap.cc:654] T 7b3ad3ddd4bb4376b0591449f3fae42f P b5cb93cc116b4642bbaa4356a44d5e19: Neither blocks nor log segments found. Creating new log.
I20260620 14:52:21.442214 19659 log.cc:824] T 7b3ad3ddd4bb4376b0591449f3fae42f P b5cb93cc116b4642bbaa4356a44d5e19: Log is configured to *not* fsync() on all Append() calls
I20260620 14:52:21.444031 19660 tablet_bootstrap.cc:492] T 7b3ad3ddd4bb4376b0591449f3fae42f P a048f73b63c14ea38806d99c64a644c1: Bootstrap starting.
I20260620 14:52:21.445173 19661 tablet_bootstrap.cc:654] T 7b3ad3ddd4bb4376b0591449f3fae42f P f2ea51c4c5ac49bc8ab03373169895ae: Neither blocks nor log segments found. Creating new log.
I20260620 14:52:21.446549 19659 tablet_bootstrap.cc:492] T 7b3ad3ddd4bb4376b0591449f3fae42f P b5cb93cc116b4642bbaa4356a44d5e19: No bootstrap required, opened a new log
I20260620 14:52:21.446925 19659 ts_tablet_manager.cc:1397] T 7b3ad3ddd4bb4376b0591449f3fae42f P b5cb93cc116b4642bbaa4356a44d5e19: Time spent bootstrapping tablet: real 0.011s user 0.009s sys 0.000s
I20260620 14:52:21.446888 19661 log.cc:824] T 7b3ad3ddd4bb4376b0591449f3fae42f P f2ea51c4c5ac49bc8ab03373169895ae: Log is configured to *not* fsync() on all Append() calls
I20260620 14:52:21.448280 19660 tablet_bootstrap.cc:654] T 7b3ad3ddd4bb4376b0591449f3fae42f P a048f73b63c14ea38806d99c64a644c1: Neither blocks nor log segments found. Creating new log.
I20260620 14:52:21.449751 19660 log.cc:824] T 7b3ad3ddd4bb4376b0591449f3fae42f P a048f73b63c14ea38806d99c64a644c1: Log is configured to *not* fsync() on all Append() calls
I20260620 14:52:21.450821 19661 tablet_bootstrap.cc:492] T 7b3ad3ddd4bb4376b0591449f3fae42f P f2ea51c4c5ac49bc8ab03373169895ae: No bootstrap required, opened a new log
I20260620 14:52:21.451161 19661 ts_tablet_manager.cc:1397] T 7b3ad3ddd4bb4376b0591449f3fae42f P f2ea51c4c5ac49bc8ab03373169895ae: Time spent bootstrapping tablet: real 0.011s user 0.003s sys 0.004s
I20260620 14:52:21.453621 19660 tablet_bootstrap.cc:492] T 7b3ad3ddd4bb4376b0591449f3fae42f P a048f73b63c14ea38806d99c64a644c1: No bootstrap required, opened a new log
I20260620 14:52:21.453994 19660 ts_tablet_manager.cc:1397] T 7b3ad3ddd4bb4376b0591449f3fae42f P a048f73b63c14ea38806d99c64a644c1: Time spent bootstrapping tablet: real 0.010s user 0.007s sys 0.000s
I20260620 14:52:21.455497 19659 raft_consensus.cc:348] T 7b3ad3ddd4bb4376b0591449f3fae42f P b5cb93cc116b4642bbaa4356a44d5e19 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b5cb93cc116b4642bbaa4356a44d5e19" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 43201 } } peers { permanent_uuid: "f2ea51c4c5ac49bc8ab03373169895ae" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 42301 } } peers { permanent_uuid: "a048f73b63c14ea38806d99c64a644c1" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 36173 } }
I20260620 14:52:21.456106 19659 raft_consensus.cc:374] T 7b3ad3ddd4bb4376b0591449f3fae42f P b5cb93cc116b4642bbaa4356a44d5e19 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 14:52:21.456216 19659 raft_consensus.cc:729] T 7b3ad3ddd4bb4376b0591449f3fae42f P b5cb93cc116b4642bbaa4356a44d5e19 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b5cb93cc116b4642bbaa4356a44d5e19, State: Initialized, Role: FOLLOWER
I20260620 14:52:21.456902 19659 consensus_queue.cc:260] T 7b3ad3ddd4bb4376b0591449f3fae42f P b5cb93cc116b4642bbaa4356a44d5e19 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b5cb93cc116b4642bbaa4356a44d5e19" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 43201 } } peers { permanent_uuid: "f2ea51c4c5ac49bc8ab03373169895ae" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 42301 } } peers { permanent_uuid: "a048f73b63c14ea38806d99c64a644c1" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 36173 } }
I20260620 14:52:21.458846 19659 ts_tablet_manager.cc:1428] T 7b3ad3ddd4bb4376b0591449f3fae42f P b5cb93cc116b4642bbaa4356a44d5e19: Time spent starting tablet: real 0.012s user 0.011s sys 0.000s
I20260620 14:52:21.464284 19661 raft_consensus.cc:348] T 7b3ad3ddd4bb4376b0591449f3fae42f P f2ea51c4c5ac49bc8ab03373169895ae [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b5cb93cc116b4642bbaa4356a44d5e19" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 43201 } } peers { permanent_uuid: "f2ea51c4c5ac49bc8ab03373169895ae" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 42301 } } peers { permanent_uuid: "a048f73b63c14ea38806d99c64a644c1" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 36173 } }
I20260620 14:52:21.464882 19661 raft_consensus.cc:374] T 7b3ad3ddd4bb4376b0591449f3fae42f P f2ea51c4c5ac49bc8ab03373169895ae [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 14:52:21.465011 19661 raft_consensus.cc:729] T 7b3ad3ddd4bb4376b0591449f3fae42f P f2ea51c4c5ac49bc8ab03373169895ae [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f2ea51c4c5ac49bc8ab03373169895ae, State: Initialized, Role: FOLLOWER
I20260620 14:52:21.465752 19661 consensus_queue.cc:260] T 7b3ad3ddd4bb4376b0591449f3fae42f P f2ea51c4c5ac49bc8ab03373169895ae [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b5cb93cc116b4642bbaa4356a44d5e19" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 43201 } } peers { permanent_uuid: "f2ea51c4c5ac49bc8ab03373169895ae" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 42301 } } peers { permanent_uuid: "a048f73b63c14ea38806d99c64a644c1" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 36173 } }
I20260620 14:52:21.466140 19660 raft_consensus.cc:348] T 7b3ad3ddd4bb4376b0591449f3fae42f P a048f73b63c14ea38806d99c64a644c1 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b5cb93cc116b4642bbaa4356a44d5e19" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 43201 } } peers { permanent_uuid: "f2ea51c4c5ac49bc8ab03373169895ae" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 42301 } } peers { permanent_uuid: "a048f73b63c14ea38806d99c64a644c1" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 36173 } }
I20260620 14:52:21.466657 19660 raft_consensus.cc:374] T 7b3ad3ddd4bb4376b0591449f3fae42f P a048f73b63c14ea38806d99c64a644c1 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 14:52:21.466777 19660 raft_consensus.cc:729] T 7b3ad3ddd4bb4376b0591449f3fae42f P a048f73b63c14ea38806d99c64a644c1 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a048f73b63c14ea38806d99c64a644c1, State: Initialized, Role: FOLLOWER
I20260620 14:52:21.467041 19511 heartbeater.cc:503] Master 127.0.97.62:32963 was elected leader, sending a full tablet report...
I20260620 14:52:21.467516 19660 consensus_queue.cc:260] T 7b3ad3ddd4bb4376b0591449f3fae42f P a048f73b63c14ea38806d99c64a644c1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b5cb93cc116b4642bbaa4356a44d5e19" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 43201 } } peers { permanent_uuid: "f2ea51c4c5ac49bc8ab03373169895ae" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 42301 } } peers { permanent_uuid: "a048f73b63c14ea38806d99c64a644c1" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 36173 } }
I20260620 14:52:21.467973 19661 ts_tablet_manager.cc:1428] T 7b3ad3ddd4bb4376b0591449f3fae42f P f2ea51c4c5ac49bc8ab03373169895ae: Time spent starting tablet: real 0.017s user 0.010s sys 0.008s
I20260620 14:52:21.469091 19641 heartbeater.cc:503] Master 127.0.97.62:32963 was elected leader, sending a full tablet report...
I20260620 14:52:21.469611 19660 ts_tablet_manager.cc:1428] T 7b3ad3ddd4bb4376b0591449f3fae42f P a048f73b63c14ea38806d99c64a644c1: Time spent starting tablet: real 0.015s user 0.013s sys 0.004s
I20260620 14:52:21.538331 19666 raft_consensus.cc:482] T 7b3ad3ddd4bb4376b0591449f3fae42f P f2ea51c4c5ac49bc8ab03373169895ae [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260620 14:52:21.538638 19666 raft_consensus.cc:504] T 7b3ad3ddd4bb4376b0591449f3fae42f P f2ea51c4c5ac49bc8ab03373169895ae [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b5cb93cc116b4642bbaa4356a44d5e19" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 43201 } } peers { permanent_uuid: "f2ea51c4c5ac49bc8ab03373169895ae" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 42301 } } peers { permanent_uuid: "a048f73b63c14ea38806d99c64a644c1" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 36173 } }
I20260620 14:52:21.540453 19666 leader_election.cc:288] T 7b3ad3ddd4bb4376b0591449f3fae42f P f2ea51c4c5ac49bc8ab03373169895ae [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers b5cb93cc116b4642bbaa4356a44d5e19 (127.0.97.1:43201), a048f73b63c14ea38806d99c64a644c1 (127.0.97.3:36173)
I20260620 14:52:21.547644 19657 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:52:21.541578 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.2:59959 (local address 127.0.97.3:36173)
0620 14:52:21.541789 (+ 211us) server_negotiation.cc:206] Beginning negotiation
0620 14:52:21.541797 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0620 14:52:21.542158 (+ 361us) server_negotiation.cc:407] Connection header received
0620 14:52:21.542414 (+ 256us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:52:21.542418 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:52:21.542480 (+ 62us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:52:21.542595 (+ 115us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 14:52:21.543187 (+ 592us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:52:21.543978 (+ 791us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:52:21.545915 (+ 1937us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:52:21.546687 (+ 772us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:52:21.546840 (+ 153us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:52:21.546948 (+ 108us) server_negotiation.cc:1035] Waiting for connection context
0620 14:52:21.547251 (+ 303us) server_negotiation.cc:299] Negotiation successful
0620 14:52:21.547374 (+ 123us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":56}
I20260620 14:52:21.547751 19669 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:52:21.540749 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.3:36173 (local address 127.0.97.2:59959)
0620 14:52:21.542015 (+ 1266us) negotiation.cc:107] Waiting for socket to connect
0620 14:52:21.542040 (+ 25us) client_negotiation.cc:174] Beginning negotiation
0620 14:52:21.542253 (+ 213us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 14:52:21.542620 (+ 367us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 14:52:21.542625 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 14:52:21.542650 (+ 25us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 14:52:21.543028 (+ 378us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:52:21.543040 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:52:21.544115 (+ 1075us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:52:21.544128 (+ 13us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:52:21.545719 (+ 1591us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:52:21.545732 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:52:21.546890 (+ 1158us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:52:21.546897 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:52:21.547059 (+ 162us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:52:21.547082 (+ 23us) client_negotiation.cc:769] Sending connection context
0620 14:52:21.547193 (+ 111us) client_negotiation.cc:240] Negotiation successful
0620 14:52:21.547337 (+ 144us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1098,"mutex_wait_us":62,"thread_start_us":1136,"threads_started":1}
I20260620 14:52:21.548091 19654 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:52:21.540987 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.2:58659 (local address 127.0.97.1:43201)
0620 14:52:21.541438 (+ 451us) server_negotiation.cc:206] Beginning negotiation
0620 14:52:21.541448 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0620 14:52:21.542339 (+ 891us) server_negotiation.cc:407] Connection header received
0620 14:52:21.542567 (+ 228us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:52:21.542572 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:52:21.542652 (+ 80us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:52:21.542776 (+ 124us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 14:52:21.543442 (+ 666us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:52:21.544260 (+ 818us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:52:21.546209 (+ 1949us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:52:21.547076 (+ 867us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:52:21.547240 (+ 164us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:52:21.547360 (+ 120us) server_negotiation.cc:1035] Waiting for connection context
0620 14:52:21.547693 (+ 333us) server_negotiation.cc:299] Negotiation successful
0620 14:52:21.547833 (+ 140us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":225}
I20260620 14:52:21.548173 19668 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:52:21.540960 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:43201 (local address 127.0.97.2:58659)
0620 14:52:21.542179 (+ 1219us) negotiation.cc:107] Waiting for socket to connect
0620 14:52:21.542196 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0620 14:52:21.542380 (+ 184us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 14:52:21.542801 (+ 421us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 14:52:21.542806 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 14:52:21.542844 (+ 38us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 14:52:21.543257 (+ 413us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:52:21.543267 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:52:21.544420 (+ 1153us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:52:21.544432 (+ 12us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:52:21.546010 (+ 1578us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:52:21.546022 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:52:21.547286 (+ 1264us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:52:21.547293 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:52:21.547477 (+ 184us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:52:21.547500 (+ 23us) client_negotiation.cc:769] Sending connection context
0620 14:52:21.547639 (+ 139us) client_negotiation.cc:240] Negotiation successful
0620 14:52:21.547791 (+ 152us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1109,"mutex_wait_us":24,"thread_start_us":680,"threads_started":1}
I20260620 14:52:21.549072 19596 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7b3ad3ddd4bb4376b0591449f3fae42f" candidate_uuid: "f2ea51c4c5ac49bc8ab03373169895ae" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a048f73b63c14ea38806d99c64a644c1" is_pre_election: true
I20260620 14:52:21.549214 19335 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7b3ad3ddd4bb4376b0591449f3fae42f" candidate_uuid: "f2ea51c4c5ac49bc8ab03373169895ae" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b5cb93cc116b4642bbaa4356a44d5e19" is_pre_election: true
I20260620 14:52:21.549731 19596 raft_consensus.cc:2454] T 7b3ad3ddd4bb4376b0591449f3fae42f P a048f73b63c14ea38806d99c64a644c1 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f2ea51c4c5ac49bc8ab03373169895ae in term 0.
I20260620 14:52:21.549731 19335 raft_consensus.cc:2454] T 7b3ad3ddd4bb4376b0591449f3fae42f P b5cb93cc116b4642bbaa4356a44d5e19 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f2ea51c4c5ac49bc8ab03373169895ae in term 0.
I20260620 14:52:21.550580 19400 leader_election.cc:302] T 7b3ad3ddd4bb4376b0591449f3fae42f P f2ea51c4c5ac49bc8ab03373169895ae [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: b5cb93cc116b4642bbaa4356a44d5e19, f2ea51c4c5ac49bc8ab03373169895ae; no voters:
I20260620 14:52:21.551054 19666 raft_consensus.cc:2781] T 7b3ad3ddd4bb4376b0591449f3fae42f P f2ea51c4c5ac49bc8ab03373169895ae [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260620 14:52:21.551205 19666 raft_consensus.cc:482] T 7b3ad3ddd4bb4376b0591449f3fae42f P f2ea51c4c5ac49bc8ab03373169895ae [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260620 14:52:21.551314 19666 raft_consensus.cc:3037] T 7b3ad3ddd4bb4376b0591449f3fae42f P f2ea51c4c5ac49bc8ab03373169895ae [term 0 FOLLOWER]: Advancing to term 1
I20260620 14:52:21.553678 19666 raft_consensus.cc:504] T 7b3ad3ddd4bb4376b0591449f3fae42f P f2ea51c4c5ac49bc8ab03373169895ae [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b5cb93cc116b4642bbaa4356a44d5e19" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 43201 } } peers { permanent_uuid: "f2ea51c4c5ac49bc8ab03373169895ae" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 42301 } } peers { permanent_uuid: "a048f73b63c14ea38806d99c64a644c1" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 36173 } }
I20260620 14:52:21.554430 19666 leader_election.cc:288] T 7b3ad3ddd4bb4376b0591449f3fae42f P f2ea51c4c5ac49bc8ab03373169895ae [CANDIDATE]: Term 1 election: Requested vote from peers b5cb93cc116b4642bbaa4356a44d5e19 (127.0.97.1:43201), a048f73b63c14ea38806d99c64a644c1 (127.0.97.3:36173)
I20260620 14:52:21.554840 19335 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7b3ad3ddd4bb4376b0591449f3fae42f" candidate_uuid: "f2ea51c4c5ac49bc8ab03373169895ae" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b5cb93cc116b4642bbaa4356a44d5e19"
I20260620 14:52:21.555018 19596 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7b3ad3ddd4bb4376b0591449f3fae42f" candidate_uuid: "f2ea51c4c5ac49bc8ab03373169895ae" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a048f73b63c14ea38806d99c64a644c1"
I20260620 14:52:21.555105 19335 raft_consensus.cc:3037] T 7b3ad3ddd4bb4376b0591449f3fae42f P b5cb93cc116b4642bbaa4356a44d5e19 [term 0 FOLLOWER]: Advancing to term 1
I20260620 14:52:21.555292 19596 raft_consensus.cc:3037] T 7b3ad3ddd4bb4376b0591449f3fae42f P a048f73b63c14ea38806d99c64a644c1 [term 0 FOLLOWER]: Advancing to term 1
I20260620 14:52:21.557601 19335 raft_consensus.cc:2454] T 7b3ad3ddd4bb4376b0591449f3fae42f P b5cb93cc116b4642bbaa4356a44d5e19 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f2ea51c4c5ac49bc8ab03373169895ae in term 1.
I20260620 14:52:21.557606 19596 raft_consensus.cc:2454] T 7b3ad3ddd4bb4376b0591449f3fae42f P a048f73b63c14ea38806d99c64a644c1 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f2ea51c4c5ac49bc8ab03373169895ae in term 1.
I20260620 14:52:21.558132 19400 leader_election.cc:302] T 7b3ad3ddd4bb4376b0591449f3fae42f P f2ea51c4c5ac49bc8ab03373169895ae [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: b5cb93cc116b4642bbaa4356a44d5e19, f2ea51c4c5ac49bc8ab03373169895ae; no voters:
I20260620 14:52:21.558446 19666 raft_consensus.cc:2781] T 7b3ad3ddd4bb4376b0591449f3fae42f P f2ea51c4c5ac49bc8ab03373169895ae [term 1 FOLLOWER]: Leader election won for term 1
I20260620 14:52:21.558857 19666 raft_consensus.cc:686] T 7b3ad3ddd4bb4376b0591449f3fae42f P f2ea51c4c5ac49bc8ab03373169895ae [term 1 LEADER]: Becoming Leader. State: Replica: f2ea51c4c5ac49bc8ab03373169895ae, State: Running, Role: LEADER
I20260620 14:52:21.559363 19666 consensus_queue.cc:237] T 7b3ad3ddd4bb4376b0591449f3fae42f P f2ea51c4c5ac49bc8ab03373169895ae [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b5cb93cc116b4642bbaa4356a44d5e19" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 43201 } } peers { permanent_uuid: "f2ea51c4c5ac49bc8ab03373169895ae" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 42301 } } peers { permanent_uuid: "a048f73b63c14ea38806d99c64a644c1" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 36173 } }
I20260620 14:52:21.564612 19194 catalog_manager.cc:5495] T 7b3ad3ddd4bb4376b0591449f3fae42f P f2ea51c4c5ac49bc8ab03373169895ae reported cstate change: term changed from 0 to 1, leader changed from <none> to f2ea51c4c5ac49bc8ab03373169895ae (127.0.97.2). New cstate: current_term: 1 leader_uuid: "f2ea51c4c5ac49bc8ab03373169895ae" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b5cb93cc116b4642bbaa4356a44d5e19" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 43201 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "f2ea51c4c5ac49bc8ab03373169895ae" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 42301 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "a048f73b63c14ea38806d99c64a644c1" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 36173 } health_report { overall_health: UNKNOWN } } }
W20260620 14:52:21.573388 19642 tablet.cc:2369] T 7b3ad3ddd4bb4376b0591449f3fae42f P a048f73b63c14ea38806d99c64a644c1: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260620 14:52:21.591562 19512 tablet.cc:2369] T 7b3ad3ddd4bb4376b0591449f3fae42f P f2ea51c4c5ac49bc8ab03373169895ae: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260620 14:52:21.602255 19656 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:52:21.596482 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38456 (local address 127.0.97.2:42301)
0620 14:52:21.596702 (+ 220us) server_negotiation.cc:206] Beginning negotiation
0620 14:52:21.596708 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 14:52:21.596778 (+ 70us) server_negotiation.cc:407] Connection header received
0620 14:52:21.597022 (+ 244us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:52:21.597026 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:52:21.597115 (+ 89us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:52:21.597270 (+ 155us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 14:52:21.597649 (+ 379us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:52:21.598354 (+ 705us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:52:21.599825 (+ 1471us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:52:21.600257 (+ 432us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:52:21.600401 (+ 144us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:52:21.601078 (+ 677us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 14:52:21.601622 (+ 544us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0620 14:52:21.601818 (+ 196us) server_negotiation.cc:1035] Waiting for connection context
0620 14:52:21.601948 (+ 130us) server_negotiation.cc:299] Negotiation successful
0620 14:52:21.601993 (+ 45us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":80}
I20260620 14:52:21.611682 19596 raft_consensus.cc:1261] T 7b3ad3ddd4bb4376b0591449f3fae42f P a048f73b63c14ea38806d99c64a644c1 [term 1 FOLLOWER]: Refusing update from remote peer f2ea51c4c5ac49bc8ab03373169895ae: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 14:52:21.611680 19335 raft_consensus.cc:1261] T 7b3ad3ddd4bb4376b0591449f3fae42f P b5cb93cc116b4642bbaa4356a44d5e19 [term 1 FOLLOWER]: Refusing update from remote peer f2ea51c4c5ac49bc8ab03373169895ae: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 14:52:21.612985 19670 consensus_queue.cc:1035] T 7b3ad3ddd4bb4376b0591449f3fae42f P f2ea51c4c5ac49bc8ab03373169895ae [LEADER]: Connected to new peer: Peer: permanent_uuid: "b5cb93cc116b4642bbaa4356a44d5e19" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 43201 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 14:52:21.613427 19666 consensus_queue.cc:1035] T 7b3ad3ddd4bb4376b0591449f3fae42f P f2ea51c4c5ac49bc8ab03373169895ae [LEADER]: Connected to new peer: Peer: permanent_uuid: "a048f73b63c14ea38806d99c64a644c1" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 36173 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20260620 14:52:21.619827 19381 tablet.cc:2369] T 7b3ad3ddd4bb4376b0591449f3fae42f P b5cb93cc116b4642bbaa4356a44d5e19: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260620 14:52:21.647202 19193 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:35966:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 14:52:21.647528 19193 catalog_manager.cc:2645] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:35966:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 14:52:21.651515 19193 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 809bcdc911834f1a98bf8e28ad2a41c4: Sending DeleteTablet for 3 replicas of tablet 7b3ad3ddd4bb4376b0591449f3fae42f
I20260620 14:52:21.652935 19315 tablet_service.cc:1514] Processing DeleteTablet for tablet 7b3ad3ddd4bb4376b0591449f3fae42f with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 14:52:21 UTC) from {username='slave'} at 127.0.0.1:37260
I20260620 14:52:21.653066 19446 tablet_service.cc:1514] Processing DeleteTablet for tablet 7b3ad3ddd4bb4376b0591449f3fae42f with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 14:52:21 UTC) from {username='slave'} at 127.0.0.1:38440
I20260620 14:52:21.653183 19576 tablet_service.cc:1514] Processing DeleteTablet for tablet 7b3ad3ddd4bb4376b0591449f3fae42f with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 14:52:21 UTC) from {username='slave'} at 127.0.0.1:39308
I20260620 14:52:21.655326 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskUci3DU/build/asan/bin/kudu pid 19253
I20260620 14:52:21.670775 19683 tablet_replica.cc:330] T 7b3ad3ddd4bb4376b0591449f3fae42f P b5cb93cc116b4642bbaa4356a44d5e19: stopping tablet replica
I20260620 14:52:21.671084 19685 tablet_replica.cc:330] T 7b3ad3ddd4bb4376b0591449f3fae42f P a048f73b63c14ea38806d99c64a644c1: stopping tablet replica
I20260620 14:52:21.671378 19684 tablet_replica.cc:330] T 7b3ad3ddd4bb4376b0591449f3fae42f P f2ea51c4c5ac49bc8ab03373169895ae: stopping tablet replica
I20260620 14:52:21.671422 19683 raft_consensus.cc:2229] T 7b3ad3ddd4bb4376b0591449f3fae42f P b5cb93cc116b4642bbaa4356a44d5e19 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 14:52:21.671794 19684 raft_consensus.cc:2229] T 7b3ad3ddd4bb4376b0591449f3fae42f P f2ea51c4c5ac49bc8ab03373169895ae [term 1 LEADER]: Raft consensus shutting down.
I20260620 14:52:21.671783 19685 raft_consensus.cc:2229] T 7b3ad3ddd4bb4376b0591449f3fae42f P a048f73b63c14ea38806d99c64a644c1 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 14:52:21.672194 19683 raft_consensus.cc:2258] T 7b3ad3ddd4bb4376b0591449f3fae42f P b5cb93cc116b4642bbaa4356a44d5e19 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 14:52:21.672272 19685 raft_consensus.cc:2258] T 7b3ad3ddd4bb4376b0591449f3fae42f P a048f73b63c14ea38806d99c64a644c1 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 14:52:21.672559 19684 raft_consensus.cc:2258] T 7b3ad3ddd4bb4376b0591449f3fae42f P f2ea51c4c5ac49bc8ab03373169895ae [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 14:52:21.674921 19683 ts_tablet_manager.cc:1905] T 7b3ad3ddd4bb4376b0591449f3fae42f P b5cb93cc116b4642bbaa4356a44d5e19: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 14:52:21.675274 19685 ts_tablet_manager.cc:1905] T 7b3ad3ddd4bb4376b0591449f3fae42f P a048f73b63c14ea38806d99c64a644c1: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 14:52:21.677525 19654 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:52:21.660502 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37272 (local address 127.0.97.1:43201)
0620 14:52:21.660754 (+ 252us) server_negotiation.cc:206] Beginning negotiation
0620 14:52:21.660764 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0620 14:52:21.662456 (+ 1692us) server_negotiation.cc:407] Connection header received
0620 14:52:21.663464 (+ 1008us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:52:21.663471 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:52:21.663636 (+ 165us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:52:21.663733 (+ 97us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:52:21.665043 (+ 1310us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:52:21.666296 (+ 1253us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:52:21.672931 (+ 6635us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:52:21.673683 (+ 752us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:52:21.673811 (+ 128us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:52:21.675561 (+ 1750us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 14:52:21.675574 (+ 13us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 14:52:21.675591 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 14:52:21.676484 (+ 893us) server_negotiation.cc:922] calling sasl_server_start()
0620 14:52:21.676552 (+ 68us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 14:52:21.676584 (+ 32us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 14:52:21.676691 (+ 107us) server_negotiation.cc:1035] Waiting for connection context
0620 14:52:21.676979 (+ 288us) server_negotiation.cc:299] Negotiation successful
0620 14:52:21.677117 (+ 138us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":84}
I20260620 14:52:21.678740 19375 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 14:52:21.681308 19684 ts_tablet_manager.cc:1905] T 7b3ad3ddd4bb4376b0591449f3fae42f P f2ea51c4c5ac49bc8ab03373169895ae: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 14:52:21.687354 19685 ts_tablet_manager.cc:1918] T 7b3ad3ddd4bb4376b0591449f3fae42f P a048f73b63c14ea38806d99c64a644c1: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260620 14:52:21.687834 19685 log.cc:1196] T 7b3ad3ddd4bb4376b0591449f3fae42f P a048f73b63c14ea38806d99c64a644c1: Deleting WAL directory at /tmp/dist-test-taskUci3DU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781966829091300-388-0/minicluster-data/ts-2/wal/wals/7b3ad3ddd4bb4376b0591449f3fae42f
I20260620 14:52:21.688803 19685 ts_tablet_manager.cc:1939] T 7b3ad3ddd4bb4376b0591449f3fae42f P a048f73b63c14ea38806d99c64a644c1: Deleting consensus metadata
I20260620 14:52:21.689531 19684 ts_tablet_manager.cc:1918] T 7b3ad3ddd4bb4376b0591449f3fae42f P f2ea51c4c5ac49bc8ab03373169895ae: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260620 14:52:21.689800 19684 log.cc:1196] T 7b3ad3ddd4bb4376b0591449f3fae42f P f2ea51c4c5ac49bc8ab03373169895ae: Deleting WAL directory at /tmp/dist-test-taskUci3DU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781966829091300-388-0/minicluster-data/ts-1/wal/wals/7b3ad3ddd4bb4376b0591449f3fae42f
I20260620 14:52:21.690420 19684 ts_tablet_manager.cc:1939] T 7b3ad3ddd4bb4376b0591449f3fae42f P f2ea51c4c5ac49bc8ab03373169895ae: Deleting consensus metadata
I20260620 14:52:21.691871 19181 catalog_manager.cc:4841] TS a048f73b63c14ea38806d99c64a644c1 (127.0.97.3:36173): tablet 7b3ad3ddd4bb4376b0591449f3fae42f (table test-table [id=78ab1d797ff741d7a7f98ed94d0123bf]) successfully deleted
I20260620 14:52:21.692337 19181 catalog_manager.cc:4841] TS f2ea51c4c5ac49bc8ab03373169895ae (127.0.97.2:42301): tablet 7b3ad3ddd4bb4376b0591449f3fae42f (table test-table [id=78ab1d797ff741d7a7f98ed94d0123bf]) successfully deleted
I20260620 14:52:21.786374 19683 ts_tablet_manager.cc:1918] T 7b3ad3ddd4bb4376b0591449f3fae42f P b5cb93cc116b4642bbaa4356a44d5e19: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260620 14:52:21.786620 19683 log.cc:1196] T 7b3ad3ddd4bb4376b0591449f3fae42f P b5cb93cc116b4642bbaa4356a44d5e19: Deleting WAL directory at /tmp/dist-test-taskUci3DU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781966829091300-388-0/minicluster-data/ts-0/wal/wals/7b3ad3ddd4bb4376b0591449f3fae42f
I20260620 14:52:21.787160 19683 ts_tablet_manager.cc:1939] T 7b3ad3ddd4bb4376b0591449f3fae42f P b5cb93cc116b4642bbaa4356a44d5e19: Deleting consensus metadata
I20260620 14:52:21.789384 19179 catalog_manager.cc:4841] TS b5cb93cc116b4642bbaa4356a44d5e19 (127.0.97.1:43201): tablet 7b3ad3ddd4bb4376b0591449f3fae42f (table test-table [id=78ab1d797ff741d7a7f98ed94d0123bf]) successfully deleted
I20260620 14:52:21.789978 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskUci3DU/build/asan/bin/kudu with pid 19253
I20260620 14:52:21.809542 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskUci3DU/build/asan/bin/kudu pid 19384
I20260620 14:52:21.817291 19656 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:52:21.810507 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38458 (local address 127.0.97.2:42301)
0620 14:52:21.810723 (+ 216us) server_negotiation.cc:206] Beginning negotiation
0620 14:52:21.810731 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0620 14:52:21.810754 (+ 23us) server_negotiation.cc:407] Connection header received
0620 14:52:21.810834 (+ 80us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:52:21.810847 (+ 13us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:52:21.810932 (+ 85us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:52:21.811048 (+ 116us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:52:21.811497 (+ 449us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:52:21.812381 (+ 884us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:52:21.813752 (+ 1371us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:52:21.814216 (+ 464us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:52:21.814358 (+ 142us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:52:21.815640 (+ 1282us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 14:52:21.815657 (+ 17us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 14:52:21.815684 (+ 27us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 14:52:21.816444 (+ 760us) server_negotiation.cc:922] calling sasl_server_start()
0620 14:52:21.816500 (+ 56us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 14:52:21.816521 (+ 21us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 14:52:21.816648 (+ 127us) server_negotiation.cc:1035] Waiting for connection context
0620 14:52:21.816796 (+ 148us) server_negotiation.cc:299] Negotiation successful
0620 14:52:21.816906 (+ 110us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":81}
I20260620 14:52:21.818272 19506 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 14:52:21.935427 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskUci3DU/build/asan/bin/kudu with pid 19384
I20260620 14:52:21.954460 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskUci3DU/build/asan/bin/kudu pid 19514
I20260620 14:52:21.961603 19657 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:52:21.955226 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39312 (local address 127.0.97.3:36173)
0620 14:52:21.955449 (+ 223us) server_negotiation.cc:206] Beginning negotiation
0620 14:52:21.955456 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0620 14:52:21.955534 (+ 78us) server_negotiation.cc:407] Connection header received
0620 14:52:21.955812 (+ 278us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:52:21.955818 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:52:21.955912 (+ 94us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:52:21.956026 (+ 114us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:52:21.956455 (+ 429us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:52:21.957193 (+ 738us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:52:21.958521 (+ 1328us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:52:21.958939 (+ 418us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:52:21.959080 (+ 141us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:52:21.960277 (+ 1197us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 14:52:21.960286 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 14:52:21.960301 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 14:52:21.960917 (+ 616us) server_negotiation.cc:922] calling sasl_server_start()
0620 14:52:21.960971 (+ 54us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 14:52:21.960992 (+ 21us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 14:52:21.961090 (+ 98us) server_negotiation.cc:1035] Waiting for connection context
0620 14:52:21.961217 (+ 127us) server_negotiation.cc:299] Negotiation successful
0620 14:52:21.961328 (+ 111us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":64}
I20260620 14:52:21.962271 19636 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 14:52:22.052474 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskUci3DU/build/asan/bin/kudu with pid 19514
I20260620 14:52:22.069226 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskUci3DU/build/asan/bin/kudu pid 19163
I20260620 14:52:22.070358 19224 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 14:52:22.182257 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskUci3DU/build/asan/bin/kudu with pid 19163
2026-06-20T14:52:22Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/3 (2934 ms)
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/4
Loading random data
Initializing database '/tmp/dist-test-taskUci3DU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781966829091300-388-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 20 14:52:22 dist-test-slave-q9j6 krb5kdc[19693](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 20 14:52:22 dist-test-slave-q9j6 krb5kdc[19693](info): set up 2 sockets
Jun 20 14:52:22 dist-test-slave-q9j6 krb5kdc[19693](info): commencing operation
krb5kdc: starting...
W20260620 14:52:23.296056 388 mini_kdc.cc:121] Time spent starting KDC: real 1.071s user 0.005s sys 0.022s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskUci3DU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781966829091300-388-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskUci3DU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781966829091300-388-0/krb5kdc/test-user.keytab.
Jun 20 14:52:23 dist-test-slave-q9j6 krb5kdc[19693](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781967143, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-20T14:52:23Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T14:52:23Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.97.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskUci3DU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781966829091300-388-0/krb5kdc/kudu_127.0.97.62.keytab.
Entry for principal kudu/127.0.97.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskUci3DU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781966829091300-388-0/krb5kdc/kudu_127.0.97.62.keytab.
I20260620 14:52:23.559201 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskUci3DU/build/asan/bin/kudu
/tmp/dist-test-taskUci3DU/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskUci3DU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781966829091300-388-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskUci3DU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781966829091300-388-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskUci3DU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781966829091300-388-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskUci3DU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781966829091300-388-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:43229
--webserver_interface=127.0.97.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskUci3DU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781966829091300-388-0/krb5kdc/kudu_127.0.97.62.keytab
--principal=kudu/127.0.97.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:32793
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.97.62:43229
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskUci3DU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781966829091300-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskUci3DU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781966829091300-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskUci3DU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781966829091300-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 14:52:23.907752 19706 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 14:52:23.908087 19706 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 14:52:23.908267 19706 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 14:52:23.916041 19706 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 14:52:23.916170 19706 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 14:52:23.916288 19706 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 14:52:23.916392 19706 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 14:52:23.916467 19706 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 14:52:23.927081 19706 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:32793
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskUci3DU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781966829091300-388-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskUci3DU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781966829091300-388-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.97.62:43229
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskUci3DU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781966829091300-388-0/krb5kdc/kudu_127.0.97.62.keytab
--principal=kudu/127.0.97.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:43229
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskUci3DU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781966829091300-388-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.97.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskUci3DU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781966829091300-388-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 13:41:20 UTC on ccycloud.quasar-vdagoi.root.comops.site
build id 4457462
ASAN enabled
I20260620 14:52:23.928588 19706 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 14:52:23.930091 19706 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 14:52:23.943094 19711 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 14:52:23.943118 19712 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 14:52:23.943054 19714 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 14:52:23.944239 19706 server_base.cc:1034] running on GCE node
I20260620 14:52:23.945230 19706 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 14:52:23.946905 19706 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 14:52:23.948189 19706 hybrid_clock.cc:648] HybridClock initialized: now 1781967143948126 us; error 59 us; skew 500 ppm
I20260620 14:52:23.948575 19706 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 14:52:23 dist-test-slave-q9j6 krb5kdc[19693](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781967143, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 14:52:23.951990 19706 init.cc:377] Logged in from keytab as kudu/127.0.97.62@KRBTEST.COM (short username kudu)
I20260620 14:52:23.953718 19706 webserver.cc:460] Webserver started at http://127.0.97.62:39279/ using document root <none> and password file <none>
I20260620 14:52:23.954530 19706 fs_manager.cc:359] Metadata directory not provided
I20260620 14:52:23.954694 19706 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 14:52:23.955044 19706 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 14:52:23.958475 19706 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskUci3DU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781966829091300-388-0/minicluster-data/master-0/data/instance:
uuid: "01c9e681e10441b89d215dc4a02f4525"
format_stamp: "Formatted at 2026-06-20 14:52:23 on dist-test-slave-q9j6"
server_key: "696a76f52b042dee079a902d72d75bcd"
server_key_iv: "e5a133cfefa13fddcca9dac7cab76ea3"
server_key_version: "encryptionkey@0"
I20260620 14:52:23.959352 19706 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskUci3DU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781966829091300-388-0/minicluster-data/master-0/wal/instance:
uuid: "01c9e681e10441b89d215dc4a02f4525"
format_stamp: "Formatted at 2026-06-20 14:52:23 on dist-test-slave-q9j6"
server_key: "696a76f52b042dee079a902d72d75bcd"
server_key_iv: "e5a133cfefa13fddcca9dac7cab76ea3"
server_key_version: "encryptionkey@0"
I20260620 14:52:23.965484 19706 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.004s sys 0.000s
I20260620 14:52:23.969640 19721 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 14:52:23.971526 19706 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.001s sys 0.000s
I20260620 14:52:23.971805 19706 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskUci3DU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781966829091300-388-0/minicluster-data/master-0/data,/tmp/dist-test-taskUci3DU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781966829091300-388-0/minicluster-data/master-0/wal
uuid: "01c9e681e10441b89d215dc4a02f4525"
format_stamp: "Formatted at 2026-06-20 14:52:23 on dist-test-slave-q9j6"
server_key: "696a76f52b042dee079a902d72d75bcd"
server_key_iv: "e5a133cfefa13fddcca9dac7cab76ea3"
server_key_version: "encryptionkey@0"
I20260620 14:52:23.972029 19706 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskUci3DU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781966829091300-388-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskUci3DU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781966829091300-388-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskUci3DU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781966829091300-388-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 14:52:24.002260 19706 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 14:52:24.006842 19706 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 14:52:24.007202 19706 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 14:52:24.027148 19706 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.62:43229
I20260620 14:52:24.027173 19772 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.62:43229 every 8 connection(s)
I20260620 14:52:24.028831 19706 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskUci3DU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781966829091300-388-0/minicluster-data/master-0/data/info.pb
I20260620 14:52:24.032984 19773 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 14:52:24.036377 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskUci3DU/build/asan/bin/kudu as pid 19706
I20260620 14:52:24.036583 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskUci3DU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781966829091300-388-0/minicluster-data/master-0/wal/instance
I20260620 14:52:24.037160 388 external_mini_cluster.cc:1428] Setting key 43405cdf012e07c42db0ba0758fd71e7
I20260620 14:52:24.045635 19773 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 01c9e681e10441b89d215dc4a02f4525: Bootstrap starting.
I20260620 14:52:24.049041 19773 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 01c9e681e10441b89d215dc4a02f4525: Neither blocks nor log segments found. Creating new log.
I20260620 14:52:24.050144 19773 log.cc:824] T 00000000000000000000000000000000 P 01c9e681e10441b89d215dc4a02f4525: Log is configured to *not* fsync() on all Append() calls
Jun 20 14:52:24 dist-test-slave-q9j6 krb5kdc[19693](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781967143, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260620 14:52:24.053925 19773 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 01c9e681e10441b89d215dc4a02f4525: No bootstrap required, opened a new log
I20260620 14:52:24.061787 19776 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:52:24.039271 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51406 (local address 127.0.97.62:43229)
0620 14:52:24.040109 (+ 838us) server_negotiation.cc:206] Beginning negotiation
0620 14:52:24.040122 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0620 14:52:24.040164 (+ 42us) server_negotiation.cc:407] Connection header received
0620 14:52:24.041921 (+ 1757us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:52:24.041947 (+ 26us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:52:24.042400 (+ 453us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:52:24.042867 (+ 467us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:52:24.044622 (+ 1755us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:52:24.046052 (+ 1430us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:52:24.047444 (+ 1392us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:52:24.048212 (+ 768us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:52:24.048437 (+ 225us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:52:24.053570 (+ 5133us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 14:52:24.053579 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 14:52:24.053595 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 14:52:24.053636 (+ 41us) server_negotiation.cc:922] calling sasl_server_start()
0620 14:52:24.056991 (+ 3355us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:52:24.057991 (+ 1000us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:52:24.057998 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:52:24.058006 (+ 8us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:52:24.058201 (+ 195us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:52:24.058654 (+ 453us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:52:24.058658 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:52:24.058660 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:52:24.059268 (+ 608us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 14:52:24.059526 (+ 258us) server_negotiation.cc:1035] Waiting for connection context
0620 14:52:24.060077 (+ 551us) server_negotiation.cc:299] Negotiation successful
0620 14:52:24.060611 (+ 534us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":618,"thread_start_us":376,"threads_started":1}
I20260620 14:52:24.063454 19773 raft_consensus.cc:348] T 00000000000000000000000000000000 P 01c9e681e10441b89d215dc4a02f4525 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "01c9e681e10441b89d215dc4a02f4525" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 43229 } }
I20260620 14:52:24.063977 19773 raft_consensus.cc:374] T 00000000000000000000000000000000 P 01c9e681e10441b89d215dc4a02f4525 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 14:52:24.064102 19773 raft_consensus.cc:729] T 00000000000000000000000000000000 P 01c9e681e10441b89d215dc4a02f4525 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 01c9e681e10441b89d215dc4a02f4525, State: Initialized, Role: FOLLOWER
I20260620 14:52:24.064781 19773 consensus_queue.cc:260] T 00000000000000000000000000000000 P 01c9e681e10441b89d215dc4a02f4525 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "01c9e681e10441b89d215dc4a02f4525" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 43229 } }
I20260620 14:52:24.065047 19773 raft_consensus.cc:388] T 00000000000000000000000000000000 P 01c9e681e10441b89d215dc4a02f4525 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 14:52:24.065214 19773 raft_consensus.cc:482] T 00000000000000000000000000000000 P 01c9e681e10441b89d215dc4a02f4525 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 14:52:24.065388 19773 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 01c9e681e10441b89d215dc4a02f4525 [term 0 FOLLOWER]: Advancing to term 1
I20260620 14:52:24.067777 19773 raft_consensus.cc:504] T 00000000000000000000000000000000 P 01c9e681e10441b89d215dc4a02f4525 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "01c9e681e10441b89d215dc4a02f4525" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 43229 } }
I20260620 14:52:24.068392 19773 leader_election.cc:302] T 00000000000000000000000000000000 P 01c9e681e10441b89d215dc4a02f4525 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 01c9e681e10441b89d215dc4a02f4525; no voters:
I20260620 14:52:24.069017 19773 leader_election.cc:288] T 00000000000000000000000000000000 P 01c9e681e10441b89d215dc4a02f4525 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260620 14:52:24.069159 19778 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 01c9e681e10441b89d215dc4a02f4525 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 14:52:24.070679 19778 raft_consensus.cc:686] T 00000000000000000000000000000000 P 01c9e681e10441b89d215dc4a02f4525 [term 1 LEADER]: Becoming Leader. State: Replica: 01c9e681e10441b89d215dc4a02f4525, State: Running, Role: LEADER
I20260620 14:52:24.071332 19778 consensus_queue.cc:237] T 00000000000000000000000000000000 P 01c9e681e10441b89d215dc4a02f4525 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "01c9e681e10441b89d215dc4a02f4525" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 43229 } }
I20260620 14:52:24.071938 19773 sys_catalog.cc:564] T 00000000000000000000000000000000 P 01c9e681e10441b89d215dc4a02f4525 [sys.catalog]: configured and running, proceeding with master startup.
I20260620 14:52:24.074831 19779 sys_catalog.cc:455] T 00000000000000000000000000000000 P 01c9e681e10441b89d215dc4a02f4525 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "01c9e681e10441b89d215dc4a02f4525" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "01c9e681e10441b89d215dc4a02f4525" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 43229 } } }
I20260620 14:52:24.074909 19780 sys_catalog.cc:455] T 00000000000000000000000000000000 P 01c9e681e10441b89d215dc4a02f4525 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 01c9e681e10441b89d215dc4a02f4525. Latest consensus state: current_term: 1 leader_uuid: "01c9e681e10441b89d215dc4a02f4525" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "01c9e681e10441b89d215dc4a02f4525" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 43229 } } }
I20260620 14:52:24.075304 19779 sys_catalog.cc:458] T 00000000000000000000000000000000 P 01c9e681e10441b89d215dc4a02f4525 [sys.catalog]: This master's current role is: LEADER
I20260620 14:52:24.075358 19780 sys_catalog.cc:458] T 00000000000000000000000000000000 P 01c9e681e10441b89d215dc4a02f4525 [sys.catalog]: This master's current role is: LEADER
I20260620 14:52:24.081173 19785 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 14:52:24.087076 19785 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 14:52:24.098098 19785 catalog_manager.cc:1347] Generated new cluster ID: c3793afc1f2947a28a70a37aa45746f5
I20260620 14:52:24.098624 19785 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 14:52:24.132315 19785 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 14:52:24.134429 19785 catalog_manager.cc:1504] Loading token signing keys...
I20260620 14:52:24.144008 19785 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 01c9e681e10441b89d215dc4a02f4525: Generated new TSK 0
I20260620 14:52:24.145247 19785 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.0.97.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskUci3DU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781966829091300-388-0/krb5kdc/kudu_127.0.97.1.keytab.
Entry for principal kudu/127.0.97.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskUci3DU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781966829091300-388-0/krb5kdc/kudu_127.0.97.1.keytab.
I20260620 14:52:24.215652 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskUci3DU/build/asan/bin/kudu
/tmp/dist-test-taskUci3DU/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskUci3DU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781966829091300-388-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskUci3DU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781966829091300-388-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskUci3DU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781966829091300-388-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskUci3DU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781966829091300-388-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.1:0
--local_ip_for_outbound_sockets=127.0.97.1
--webserver_interface=127.0.97.1
--webserver_port=0
--tserver_master_addrs=127.0.97.62:43229
--keytab_file=/tmp/dist-test-taskUci3DU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781966829091300-388-0/krb5kdc/kudu_127.0.97.1.keytab
--principal=kudu/127.0.97.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:32793
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskUci3DU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781966829091300-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskUci3DU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781966829091300-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskUci3DU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781966829091300-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 14:52:24.573870 19799 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 14:52:24.574236 19799 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 14:52:24.574414 19799 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 14:52:24.582616 19799 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 14:52:24.582796 19799 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 14:52:24.582988 19799 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.1
I20260620 14:52:24.593498 19799 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:32793
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskUci3DU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781966829091300-388-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskUci3DU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781966829091300-388-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskUci3DU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781966829091300-388-0/krb5kdc/kudu_127.0.97.1.keytab
--principal=kudu/127.0.97.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskUci3DU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781966829091300-388-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.97.1
--webserver_port=0
--tserver_master_addrs=127.0.97.62:43229
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.1
--log_dir=/tmp/dist-test-taskUci3DU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781966829091300-388-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 13:41:20 UTC on ccycloud.quasar-vdagoi.root.comops.site
build id 4457462
ASAN enabled
I20260620 14:52:24.595006 19799 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 14:52:24.596486 19799 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 14:52:24.610555 19804 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 14:52:24.610631 19805 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 14:52:24.610589 19807 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 14:52:24.611732 19799 server_base.cc:1034] running on GCE node
I20260620 14:52:24.612413 19799 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 14:52:24.618224 19799 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 14:52:24.619557 19799 hybrid_clock.cc:648] HybridClock initialized: now 1781967144619460 us; error 88 us; skew 500 ppm
I20260620 14:52:24.620132 19799 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 14:52:24 dist-test-slave-q9j6 krb5kdc[19693](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781967144, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 14:52:24.624745 19799 init.cc:377] Logged in from keytab as kudu/127.0.97.1@KRBTEST.COM (short username kudu)
I20260620 14:52:24.630466 19799 webserver.cc:460] Webserver started at http://127.0.97.1:33981/ using document root <none> and password file <none>
I20260620 14:52:24.631495 19799 fs_manager.cc:359] Metadata directory not provided
I20260620 14:52:24.631678 19799 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 14:52:24.632020 19799 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 14:52:24.635246 19799 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskUci3DU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781966829091300-388-0/minicluster-data/ts-0/data/instance:
uuid: "64e8c290414645e8a44a84e22960df76"
format_stamp: "Formatted at 2026-06-20 14:52:24 on dist-test-slave-q9j6"
server_key: "8698f8ed9941da13f23d343d1802e9cc"
server_key_iv: "2fc17a032bc6567bdee041e597336ef0"
server_key_version: "encryptionkey@0"
I20260620 14:52:24.636170 19799 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskUci3DU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781966829091300-388-0/minicluster-data/ts-0/wal/instance:
uuid: "64e8c290414645e8a44a84e22960df76"
format_stamp: "Formatted at 2026-06-20 14:52:24 on dist-test-slave-q9j6"
server_key: "8698f8ed9941da13f23d343d1802e9cc"
server_key_iv: "2fc17a032bc6567bdee041e597336ef0"
server_key_version: "encryptionkey@0"
I20260620 14:52:24.642099 19799 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260620 14:52:24.646390 19814 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 14:52:24.648217 19799 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.002s
I20260620 14:52:24.648452 19799 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskUci3DU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781966829091300-388-0/minicluster-data/ts-0/data,/tmp/dist-test-taskUci3DU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781966829091300-388-0/minicluster-data/ts-0/wal
uuid: "64e8c290414645e8a44a84e22960df76"
format_stamp: "Formatted at 2026-06-20 14:52:24 on dist-test-slave-q9j6"
server_key: "8698f8ed9941da13f23d343d1802e9cc"
server_key_iv: "2fc17a032bc6567bdee041e597336ef0"
server_key_version: "encryptionkey@0"
I20260620 14:52:24.648691 19799 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskUci3DU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781966829091300-388-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskUci3DU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781966829091300-388-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskUci3DU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781966829091300-388-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 14:52:24.672251 19799 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 14:52:24.676729 19799 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 14:52:24.677067 19799 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 14:52:24.678100 19799 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 14:52:24.679590 19799 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 14:52:24.679756 19799 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.001s sys 0.000s
I20260620 14:52:24.679919 19799 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 14:52:24.679991 19799 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 14:52:24.714901 19799 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.1:33989
I20260620 14:52:24.714960 19926 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.1:33989 every 8 connection(s)
I20260620 14:52:24.716365 19799 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskUci3DU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781966829091300-388-0/minicluster-data/ts-0/data/info.pb
I20260620 14:52:24.719763 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskUci3DU/build/asan/bin/kudu as pid 19799
I20260620 14:52:24.719964 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskUci3DU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781966829091300-388-0/minicluster-data/ts-0/wal/instance
I20260620 14:52:24.720548 388 external_mini_cluster.cc:1428] Setting key acb2d2c7b36bf039d8171e173228c3e6
Jun 20 14:52:24 dist-test-slave-q9j6 krb5kdc[19693](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781967144, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.1@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260620 14:52:24.738632 19929 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:52:24.719051 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.1:51335 (local address 127.0.97.62:43229)
0620 14:52:24.719533 (+ 482us) server_negotiation.cc:206] Beginning negotiation
0620 14:52:24.719538 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 14:52:24.720775 (+ 1237us) server_negotiation.cc:407] Connection header received
0620 14:52:24.723074 (+ 2299us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:52:24.723079 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:52:24.723171 (+ 92us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:52:24.723307 (+ 136us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:52:24.725117 (+ 1810us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:52:24.725880 (+ 763us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:52:24.727623 (+ 1743us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:52:24.728049 (+ 426us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:52:24.728204 (+ 155us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:52:24.732342 (+ 4138us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 14:52:24.732347 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 14:52:24.732351 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 14:52:24.732390 (+ 39us) server_negotiation.cc:922] calling sasl_server_start()
0620 14:52:24.735119 (+ 2729us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:52:24.736094 (+ 975us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:52:24.736099 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:52:24.736101 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:52:24.736231 (+ 130us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:52:24.736717 (+ 486us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:52:24.736721 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:52:24.736723 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:52:24.737002 (+ 279us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 14:52:24.737169 (+ 167us) server_negotiation.cc:1035] Waiting for connection context
0620 14:52:24.737955 (+ 786us) server_negotiation.cc:299] Negotiation successful
0620 14:52:24.738230 (+ 275us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":382,"thread_start_us":259,"threads_started":1}
I20260620 14:52:24.739758 19930 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:52:24.719664 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:43229 (local address 127.0.97.1:51335)
0620 14:52:24.720559 (+ 895us) negotiation.cc:107] Waiting for socket to connect
0620 14:52:24.720626 (+ 67us) client_negotiation.cc:174] Beginning negotiation
0620 14:52:24.722764 (+ 2138us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 14:52:24.723585 (+ 821us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 14:52:24.723649 (+ 64us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 14:52:24.724294 (+ 645us) client_negotiation.cc:189] Negotiated authn=SASL
0620 14:52:24.724912 (+ 618us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:52:24.724940 (+ 28us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:52:24.726103 (+ 1163us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:52:24.726107 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:52:24.727426 (+ 1319us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:52:24.727440 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:52:24.728318 (+ 878us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:52:24.728327 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:52:24.728608 (+ 281us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:52:24.729393 (+ 785us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 14:52:24.729418 (+ 25us) client_negotiation.cc:656] calling sasl_client_start()
0620 14:52:24.732051 (+ 2633us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 14:52:24.735311 (+ 3260us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 14:52:24.735321 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 14:52:24.735344 (+ 23us) client_negotiation.cc:760] calling sasl_client_step()
0620 14:52:24.735932 (+ 588us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 14:52:24.736355 (+ 423us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 14:52:24.736361 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 14:52:24.736364 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0620 14:52:24.736583 (+ 219us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 14:52:24.737143 (+ 560us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 14:52:24.737151 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 14:52:24.737541 (+ 390us) client_negotiation.cc:769] Sending connection context
0620 14:52:24.737825 (+ 284us) client_negotiation.cc:240] Negotiation successful
0620 14:52:24.738431 (+ 606us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":621,"thread_start_us":382,"threads_started":1}
I20260620 14:52:24.741722 19927 heartbeater.cc:347] Connected to a master server at 127.0.97.62:43229
I20260620 14:52:24.742075 19927 heartbeater.cc:464] Registering TS with master...
I20260620 14:52:24.742871 19927 heartbeater.cc:511] Master 127.0.97.62:43229 requested a full tablet report, sending...
WARNING: no policy specified for kudu/127.0.97.2@KRBTEST.COM; defaulting to no policy
I20260620 14:52:24.745661 19738 ts_manager.cc:194] Registered new tserver with Master: 64e8c290414645e8a44a84e22960df76 (127.0.97.1:33989)
I20260620 14:52:24.747339 19738 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.97.1@KRBTEST.COM'} at 127.0.97.1:51335
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskUci3DU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781966829091300-388-0/krb5kdc/kudu_127.0.97.2.keytab.
Entry for principal kudu/127.0.97.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskUci3DU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781966829091300-388-0/krb5kdc/kudu_127.0.97.2.keytab.
I20260620 14:52:24.778829 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskUci3DU/build/asan/bin/kudu
/tmp/dist-test-taskUci3DU/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskUci3DU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781966829091300-388-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskUci3DU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781966829091300-388-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskUci3DU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781966829091300-388-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskUci3DU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781966829091300-388-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.2:0
--local_ip_for_outbound_sockets=127.0.97.2
--webserver_interface=127.0.97.2
--webserver_port=0
--tserver_master_addrs=127.0.97.62:43229
--keytab_file=/tmp/dist-test-taskUci3DU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781966829091300-388-0/krb5kdc/kudu_127.0.97.2.keytab
--principal=kudu/127.0.97.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:32793
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskUci3DU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781966829091300-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskUci3DU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781966829091300-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskUci3DU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781966829091300-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 14:52:25.129671 19933 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 14:52:25.129973 19933 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 14:52:25.130146 19933 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 14:52:25.137800 19933 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 14:52:25.137933 19933 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 14:52:25.138139 19933 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.2
I20260620 14:52:25.149024 19933 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:32793
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskUci3DU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781966829091300-388-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskUci3DU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781966829091300-388-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskUci3DU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781966829091300-388-0/krb5kdc/kudu_127.0.97.2.keytab
--principal=kudu/127.0.97.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskUci3DU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781966829091300-388-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.97.2
--webserver_port=0
--tserver_master_addrs=127.0.97.62:43229
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.2
--log_dir=/tmp/dist-test-taskUci3DU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781966829091300-388-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 13:41:20 UTC on ccycloud.quasar-vdagoi.root.comops.site
build id 4457462
ASAN enabled
I20260620 14:52:25.150447 19933 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 14:52:25.151917 19933 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 14:52:25.166262 19941 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 14:52:25.166236 19938 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 14:52:25.166260 19939 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 14:52:25.167220 19933 server_base.cc:1034] running on GCE node
I20260620 14:52:25.167990 19933 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 14:52:25.169032 19933 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 14:52:25.170243 19933 hybrid_clock.cc:648] HybridClock initialized: now 1781967145170170 us; error 69 us; skew 500 ppm
I20260620 14:52:25.170599 19933 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 14:52:25 dist-test-slave-q9j6 krb5kdc[19693](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781967145, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 14:52:25.174193 19933 init.cc:377] Logged in from keytab as kudu/127.0.97.2@KRBTEST.COM (short username kudu)
I20260620 14:52:25.176029 19933 webserver.cc:460] Webserver started at http://127.0.97.2:34637/ using document root <none> and password file <none>
I20260620 14:52:25.176826 19933 fs_manager.cc:359] Metadata directory not provided
I20260620 14:52:25.176985 19933 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 14:52:25.177327 19933 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 14:52:25.180544 19933 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskUci3DU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781966829091300-388-0/minicluster-data/ts-1/data/instance:
uuid: "33401df5fb2c4a818f615a24ab8c5825"
format_stamp: "Formatted at 2026-06-20 14:52:25 on dist-test-slave-q9j6"
server_key: "41c2424e4c9de1f25cc93d49e749e0ac"
server_key_iv: "3db190036c0fe4135ae83229e6504099"
server_key_version: "encryptionkey@0"
I20260620 14:52:25.181511 19933 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskUci3DU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781966829091300-388-0/minicluster-data/ts-1/wal/instance:
uuid: "33401df5fb2c4a818f615a24ab8c5825"
format_stamp: "Formatted at 2026-06-20 14:52:25 on dist-test-slave-q9j6"
server_key: "41c2424e4c9de1f25cc93d49e749e0ac"
server_key_iv: "3db190036c0fe4135ae83229e6504099"
server_key_version: "encryptionkey@0"
I20260620 14:52:25.187762 19933 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.007s sys 0.001s
I20260620 14:52:25.191726 19948 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 14:52:25.193203 19933 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260620 14:52:25.193429 19933 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskUci3DU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781966829091300-388-0/minicluster-data/ts-1/data,/tmp/dist-test-taskUci3DU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781966829091300-388-0/minicluster-data/ts-1/wal
uuid: "33401df5fb2c4a818f615a24ab8c5825"
format_stamp: "Formatted at 2026-06-20 14:52:25 on dist-test-slave-q9j6"
server_key: "41c2424e4c9de1f25cc93d49e749e0ac"
server_key_iv: "3db190036c0fe4135ae83229e6504099"
server_key_version: "encryptionkey@0"
I20260620 14:52:25.193661 19933 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskUci3DU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781966829091300-388-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskUci3DU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781966829091300-388-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskUci3DU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781966829091300-388-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 14:52:25.217882 19933 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 14:52:25.222862 19933 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 14:52:25.223192 19933 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 14:52:25.224360 19933 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 14:52:25.225801 19933 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 14:52:25.225880 19933 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 14:52:25.226056 19933 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 14:52:25.226109 19933 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 14:52:25.261970 19933 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.2:45613
I20260620 14:52:25.262041 20060 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.2:45613 every 8 connection(s)
I20260620 14:52:25.263375 19933 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskUci3DU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781966829091300-388-0/minicluster-data/ts-1/data/info.pb
I20260620 14:52:25.265523 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskUci3DU/build/asan/bin/kudu as pid 19933
I20260620 14:52:25.265717 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskUci3DU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781966829091300-388-0/minicluster-data/ts-1/wal/instance
I20260620 14:52:25.266274 388 external_mini_cluster.cc:1428] Setting key 6be8686466b7cbd876e31763cd63ca86
Jun 20 14:52:25 dist-test-slave-q9j6 krb5kdc[19693](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781967145, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.2@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260620 14:52:25.288445 20063 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:52:25.266236 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.2:51805 (local address 127.0.97.62:43229)
0620 14:52:25.267181 (+ 945us) server_negotiation.cc:206] Beginning negotiation
0620 14:52:25.267186 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 14:52:25.268188 (+ 1002us) server_negotiation.cc:407] Connection header received
0620 14:52:25.270277 (+ 2089us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:52:25.270283 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:52:25.270389 (+ 106us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:52:25.270526 (+ 137us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:52:25.272454 (+ 1928us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:52:25.273239 (+ 785us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:52:25.274907 (+ 1668us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:52:25.275361 (+ 454us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:52:25.275506 (+ 145us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:52:25.280299 (+ 4793us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 14:52:25.280309 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 14:52:25.280317 (+ 8us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 14:52:25.280370 (+ 53us) server_negotiation.cc:922] calling sasl_server_start()
0620 14:52:25.284542 (+ 4172us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:52:25.285523 (+ 981us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:52:25.285532 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:52:25.285539 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:52:25.285723 (+ 184us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:52:25.286198 (+ 475us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:52:25.286206 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:52:25.286212 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:52:25.286646 (+ 434us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 14:52:25.286819 (+ 173us) server_negotiation.cc:1035] Waiting for connection context
0620 14:52:25.287633 (+ 814us) server_negotiation.cc:299] Negotiation successful
0620 14:52:25.287926 (+ 293us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":807,"thread_start_us":674,"threads_started":1}
I20260620 14:52:25.289144 20064 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:52:25.267013 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:43229 (local address 127.0.97.2:51805)
0620 14:52:25.267993 (+ 980us) negotiation.cc:107] Waiting for socket to connect
0620 14:52:25.268051 (+ 58us) client_negotiation.cc:174] Beginning negotiation
0620 14:52:25.269955 (+ 1904us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 14:52:25.270838 (+ 883us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 14:52:25.270848 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 14:52:25.271447 (+ 599us) client_negotiation.cc:189] Negotiated authn=SASL
0620 14:52:25.272256 (+ 809us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:52:25.272277 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:52:25.273429 (+ 1152us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:52:25.273433 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:52:25.274720 (+ 1287us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:52:25.274734 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:52:25.275696 (+ 962us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:52:25.275707 (+ 11us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:52:25.275953 (+ 246us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:52:25.276790 (+ 837us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 14:52:25.276855 (+ 65us) client_negotiation.cc:656] calling sasl_client_start()
0620 14:52:25.279961 (+ 3106us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 14:52:25.284778 (+ 4817us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 14:52:25.284787 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 14:52:25.284811 (+ 24us) client_negotiation.cc:760] calling sasl_client_step()
0620 14:52:25.285340 (+ 529us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 14:52:25.285852 (+ 512us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 14:52:25.285856 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 14:52:25.285858 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 14:52:25.286062 (+ 204us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 14:52:25.286773 (+ 711us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 14:52:25.286779 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 14:52:25.287150 (+ 371us) client_negotiation.cc:769] Sending connection context
0620 14:52:25.287417 (+ 267us) client_negotiation.cc:240] Negotiation successful
0620 14:52:25.287938 (+ 521us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":655,"thread_start_us":361,"threads_started":1}
I20260620 14:52:25.290851 20061 heartbeater.cc:347] Connected to a master server at 127.0.97.62:43229
I20260620 14:52:25.291244 20061 heartbeater.cc:464] Registering TS with master...
I20260620 14:52:25.292306 20061 heartbeater.cc:511] Master 127.0.97.62:43229 requested a full tablet report, sending...
WARNING: no policy specified for kudu/127.0.97.3@KRBTEST.COM; defaulting to no policy
I20260620 14:52:25.295141 19738 ts_manager.cc:194] Registered new tserver with Master: 33401df5fb2c4a818f615a24ab8c5825 (127.0.97.2:45613)
I20260620 14:52:25.296147 19738 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.97.2@KRBTEST.COM'} at 127.0.97.2:51805
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskUci3DU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781966829091300-388-0/krb5kdc/kudu_127.0.97.3.keytab.
Entry for principal kudu/127.0.97.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskUci3DU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781966829091300-388-0/krb5kdc/kudu_127.0.97.3.keytab.
I20260620 14:52:25.328519 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskUci3DU/build/asan/bin/kudu
/tmp/dist-test-taskUci3DU/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskUci3DU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781966829091300-388-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskUci3DU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781966829091300-388-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskUci3DU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781966829091300-388-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskUci3DU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781966829091300-388-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.3:0
--local_ip_for_outbound_sockets=127.0.97.3
--webserver_interface=127.0.97.3
--webserver_port=0
--tserver_master_addrs=127.0.97.62:43229
--keytab_file=/tmp/dist-test-taskUci3DU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781966829091300-388-0/krb5kdc/kudu_127.0.97.3.keytab
--principal=kudu/127.0.97.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:32793
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskUci3DU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781966829091300-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskUci3DU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781966829091300-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskUci3DU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781966829091300-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 14:52:25.681974 20067 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 14:52:25.682345 20067 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 14:52:25.682448 20067 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 14:52:25.690092 20067 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 14:52:25.690225 20067 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 14:52:25.690347 20067 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.3
I20260620 14:52:25.701079 20067 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:32793
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskUci3DU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781966829091300-388-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskUci3DU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781966829091300-388-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskUci3DU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781966829091300-388-0/krb5kdc/kudu_127.0.97.3.keytab
--principal=kudu/127.0.97.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskUci3DU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781966829091300-388-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.97.3
--webserver_port=0
--tserver_master_addrs=127.0.97.62:43229
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.3
--log_dir=/tmp/dist-test-taskUci3DU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781966829091300-388-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 13:41:20 UTC on ccycloud.quasar-vdagoi.root.comops.site
build id 4457462
ASAN enabled
I20260620 14:52:25.702478 20067 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 14:52:25.703986 20067 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 14:52:25.719007 20073 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 14:52:25.719027 20072 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 14:52:25.719277 20075 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 14:52:25.721036 20067 server_base.cc:1034] running on GCE node
I20260620 14:52:25.721786 20067 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 14:52:25.722815 20067 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 14:52:25.724056 20067 hybrid_clock.cc:648] HybridClock initialized: now 1781967145724015 us; error 67 us; skew 500 ppm
I20260620 14:52:25.724458 20067 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 14:52:25 dist-test-slave-q9j6 krb5kdc[19693](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781967145, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 14:52:25.727941 20067 init.cc:377] Logged in from keytab as kudu/127.0.97.3@KRBTEST.COM (short username kudu)
I20260620 14:52:25.729579 20067 webserver.cc:460] Webserver started at http://127.0.97.3:42299/ using document root <none> and password file <none>
I20260620 14:52:25.730407 20067 fs_manager.cc:359] Metadata directory not provided
I20260620 14:52:25.730568 20067 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 14:52:25.730909 20067 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 14:52:25.734203 20067 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskUci3DU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781966829091300-388-0/minicluster-data/ts-2/data/instance:
uuid: "096323af948c42deb241325687a37eaa"
format_stamp: "Formatted at 2026-06-20 14:52:25 on dist-test-slave-q9j6"
server_key: "372d4b6822f57d50b5a9dc03ad68264a"
server_key_iv: "ed3b1e43904e7de874a5cc4ef8cab0df"
server_key_version: "encryptionkey@0"
I20260620 14:52:25.735119 20067 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskUci3DU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781966829091300-388-0/minicluster-data/ts-2/wal/instance:
uuid: "096323af948c42deb241325687a37eaa"
format_stamp: "Formatted at 2026-06-20 14:52:25 on dist-test-slave-q9j6"
server_key: "372d4b6822f57d50b5a9dc03ad68264a"
server_key_iv: "ed3b1e43904e7de874a5cc4ef8cab0df"
server_key_version: "encryptionkey@0"
I20260620 14:52:25.740952 20067 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.007s sys 0.000s
I20260620 14:52:25.744985 20082 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 14:52:25.746579 20067 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.001s
I20260620 14:52:25.746793 20067 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskUci3DU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781966829091300-388-0/minicluster-data/ts-2/data,/tmp/dist-test-taskUci3DU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781966829091300-388-0/minicluster-data/ts-2/wal
uuid: "096323af948c42deb241325687a37eaa"
format_stamp: "Formatted at 2026-06-20 14:52:25 on dist-test-slave-q9j6"
server_key: "372d4b6822f57d50b5a9dc03ad68264a"
server_key_iv: "ed3b1e43904e7de874a5cc4ef8cab0df"
server_key_version: "encryptionkey@0"
I20260620 14:52:25.747028 20067 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskUci3DU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781966829091300-388-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskUci3DU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781966829091300-388-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskUci3DU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781966829091300-388-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 14:52:25.750924 19927 heartbeater.cc:503] Master 127.0.97.62:43229 was elected leader, sending a full tablet report...
I20260620 14:52:25.768224 20067 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 14:52:25.773084 20067 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 14:52:25.773411 20067 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 14:52:25.774535 20067 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 14:52:25.776154 20067 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 14:52:25.776298 20067 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 14:52:25.776455 20067 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 14:52:25.776531 20067 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 14:52:25.811511 20067 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.3:39597
I20260620 14:52:25.811685 20194 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.3:39597 every 8 connection(s)
I20260620 14:52:25.812949 20067 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskUci3DU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781966829091300-388-0/minicluster-data/ts-2/data/info.pb
I20260620 14:52:25.815482 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskUci3DU/build/asan/bin/kudu as pid 20067
I20260620 14:52:25.815677 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskUci3DU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781966829091300-388-0/minicluster-data/ts-2/wal/instance
I20260620 14:52:25.816145 388 external_mini_cluster.cc:1428] Setting key 1d07614208df577a9f83f62987420c60
Jun 20 14:52:25 dist-test-slave-q9j6 krb5kdc[19693](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781967145, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.3@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260620 14:52:25.835563 20197 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:52:25.816068 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.3:41387 (local address 127.0.97.62:43229)
0620 14:52:25.816639 (+ 571us) server_negotiation.cc:206] Beginning negotiation
0620 14:52:25.816646 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0620 14:52:25.817932 (+ 1286us) server_negotiation.cc:407] Connection header received
0620 14:52:25.820240 (+ 2308us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:52:25.820246 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:52:25.820342 (+ 96us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:52:25.820474 (+ 132us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:52:25.822187 (+ 1713us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:52:25.822992 (+ 805us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:52:25.824791 (+ 1799us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:52:25.825232 (+ 441us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:52:25.825389 (+ 157us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:52:25.829078 (+ 3689us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 14:52:25.829084 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 14:52:25.829087 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 14:52:25.829124 (+ 37us) server_negotiation.cc:922] calling sasl_server_start()
0620 14:52:25.832307 (+ 3183us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:52:25.833104 (+ 797us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:52:25.833110 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:52:25.833115 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:52:25.833247 (+ 132us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:52:25.833714 (+ 467us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:52:25.833724 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:52:25.833726 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:52:25.834001 (+ 275us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 14:52:25.834163 (+ 162us) server_negotiation.cc:1035] Waiting for connection context
0620 14:52:25.834940 (+ 777us) server_negotiation.cc:299] Negotiation successful
0620 14:52:25.835179 (+ 239us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":431,"thread_start_us":270,"threads_started":1}
I20260620 14:52:25.836676 20198 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:52:25.816772 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:43229 (local address 127.0.97.3:41387)
0620 14:52:25.817771 (+ 999us) negotiation.cc:107] Waiting for socket to connect
0620 14:52:25.817819 (+ 48us) client_negotiation.cc:174] Beginning negotiation
0620 14:52:25.819937 (+ 2118us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 14:52:25.820749 (+ 812us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 14:52:25.820759 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 14:52:25.821373 (+ 614us) client_negotiation.cc:189] Negotiated authn=SASL
0620 14:52:25.821999 (+ 626us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:52:25.822018 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:52:25.823172 (+ 1154us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:52:25.823177 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:52:25.824604 (+ 1427us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:52:25.824620 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:52:25.825501 (+ 881us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:52:25.825512 (+ 11us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:52:25.825775 (+ 263us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:52:25.826449 (+ 674us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 14:52:25.826472 (+ 23us) client_negotiation.cc:656] calling sasl_client_start()
0620 14:52:25.828809 (+ 2337us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 14:52:25.832456 (+ 3647us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 14:52:25.832467 (+ 11us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 14:52:25.832488 (+ 21us) client_negotiation.cc:760] calling sasl_client_step()
0620 14:52:25.832968 (+ 480us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 14:52:25.833365 (+ 397us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 14:52:25.833370 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 14:52:25.833374 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0620 14:52:25.833585 (+ 211us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 14:52:25.834142 (+ 557us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 14:52:25.834152 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 14:52:25.834543 (+ 391us) client_negotiation.cc:769] Sending connection context
0620 14:52:25.834837 (+ 294us) client_negotiation.cc:240] Negotiation successful
0620 14:52:25.835431 (+ 594us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":645,"thread_start_us":406,"threads_started":1}
I20260620 14:52:25.838162 20195 heartbeater.cc:347] Connected to a master server at 127.0.97.62:43229
I20260620 14:52:25.838500 20195 heartbeater.cc:464] Registering TS with master...
I20260620 14:52:25.839277 20195 heartbeater.cc:511] Master 127.0.97.62:43229 requested a full tablet report, sending...
I20260620 14:52:25.840910 19738 ts_manager.cc:194] Registered new tserver with Master: 096323af948c42deb241325687a37eaa (127.0.97.3:39597)
I20260620 14:52:25.841797 19738 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.97.3@KRBTEST.COM'} at 127.0.97.3:41387
I20260620 14:52:25.844743 388 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260620 14:52:25.862608 20197 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:52:25.849153 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51414 (local address 127.0.97.62:43229)
0620 14:52:25.849389 (+ 236us) server_negotiation.cc:206] Beginning negotiation
0620 14:52:25.849400 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0620 14:52:25.849798 (+ 398us) server_negotiation.cc:407] Connection header received
0620 14:52:25.849943 (+ 145us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:52:25.849946 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:52:25.850018 (+ 72us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:52:25.850108 (+ 90us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:52:25.851853 (+ 1745us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:52:25.852671 (+ 818us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:52:25.854049 (+ 1378us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:52:25.854532 (+ 483us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:52:25.854645 (+ 113us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:52:25.857159 (+ 2514us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 14:52:25.857165 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 14:52:25.857168 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 14:52:25.857207 (+ 39us) server_negotiation.cc:922] calling sasl_server_start()
0620 14:52:25.860079 (+ 2872us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:52:25.860826 (+ 747us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:52:25.860830 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:52:25.860833 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:52:25.860969 (+ 136us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:52:25.861343 (+ 374us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:52:25.861346 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:52:25.861348 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:52:25.861684 (+ 336us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 14:52:25.861879 (+ 195us) server_negotiation.cc:1035] Waiting for connection context
0620 14:52:25.862181 (+ 302us) server_negotiation.cc:299] Negotiation successful
0620 14:52:25.862378 (+ 197us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":68}
I20260620 14:52:25.866744 19738 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:51414:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260620 14:52:25.873906 19738 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260620 14:52:25.900167 20211 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:52:25.888139 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48592 (local address 127.0.97.3:39597)
0620 14:52:25.890659 (+ 2520us) server_negotiation.cc:206] Beginning negotiation
0620 14:52:25.890675 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0620 14:52:25.890714 (+ 39us) server_negotiation.cc:407] Connection header received
0620 14:52:25.890927 (+ 213us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:52:25.890940 (+ 13us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:52:25.891351 (+ 411us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:52:25.892517 (+ 1166us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 14:52:25.892837 (+ 320us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:52:25.894564 (+ 1727us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:52:25.897501 (+ 2937us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:52:25.898776 (+ 1275us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:52:25.898947 (+ 171us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:52:25.899196 (+ 249us) server_negotiation.cc:1035] Waiting for connection context
0620 14:52:25.899556 (+ 360us) server_negotiation.cc:299] Negotiation successful
0620 14:52:25.899790 (+ 234us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":2191,"thread_start_us":3153,"threads_started":1}
I20260620 14:52:25.900204 20209 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:52:25.887540 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.3:39597 (local address 127.0.0.1:48592)
0620 14:52:25.888660 (+ 1120us) negotiation.cc:107] Waiting for socket to connect
0620 14:52:25.888714 (+ 54us) client_negotiation.cc:174] Beginning negotiation
0620 14:52:25.888896 (+ 182us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 14:52:25.891626 (+ 2730us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 14:52:25.891633 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 14:52:25.891686 (+ 53us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 14:52:25.892282 (+ 596us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:52:25.892303 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:52:25.894764 (+ 2461us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:52:25.894771 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:52:25.897322 (+ 2551us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:52:25.897336 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:52:25.898990 (+ 1654us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:52:25.898996 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:52:25.899245 (+ 249us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:52:25.899318 (+ 73us) client_negotiation.cc:769] Sending connection context
0620 14:52:25.899499 (+ 181us) client_negotiation.cc:240] Negotiation successful
0620 14:52:25.899752 (+ 253us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":858,"mutex_wait_us":115,"spinlock_wait_cycles":7808,"thread_start_us":2181,"threads_started":1}
I20260620 14:52:25.900489 20210 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:52:25.888202 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34600 (local address 127.0.97.1:33989)
0620 14:52:25.889544 (+ 1342us) server_negotiation.cc:206] Beginning negotiation
0620 14:52:25.889552 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0620 14:52:25.890114 (+ 562us) server_negotiation.cc:407] Connection header received
0620 14:52:25.890907 (+ 793us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:52:25.890920 (+ 13us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:52:25.891261 (+ 341us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:52:25.891475 (+ 214us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 14:52:25.893193 (+ 1718us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:52:25.894751 (+ 1558us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:52:25.897193 (+ 2442us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:52:25.898806 (+ 1613us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:52:25.898930 (+ 124us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:52:25.899194 (+ 264us) server_negotiation.cc:1035] Waiting for connection context
0620 14:52:25.899946 (+ 752us) server_negotiation.cc:299] Negotiation successful
0620 14:52:25.900133 (+ 187us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1186,"thread_start_us":1003,"threads_started":1}
I20260620 14:52:25.901304 20208 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:52:25.887745 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:33989 (local address 127.0.0.1:34600)
0620 14:52:25.890000 (+ 2255us) negotiation.cc:107] Waiting for socket to connect
0620 14:52:25.890025 (+ 25us) client_negotiation.cc:174] Beginning negotiation
0620 14:52:25.890686 (+ 661us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 14:52:25.891625 (+ 939us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 14:52:25.891633 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 14:52:25.891688 (+ 55us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 14:52:25.892988 (+ 1300us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:52:25.893001 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:52:25.894906 (+ 1905us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:52:25.894913 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:52:25.896958 (+ 2045us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:52:25.896980 (+ 22us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:52:25.899482 (+ 2502us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:52:25.899490 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:52:25.899710 (+ 220us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:52:25.899743 (+ 33us) client_negotiation.cc:769] Sending connection context
0620 14:52:25.900711 (+ 968us) client_negotiation.cc:240] Negotiation successful
0620 14:52:25.900878 (+ 167us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":2124,"mutex_wait_us":829,"spinlock_wait_cycles":11776,"thread_start_us":650,"threads_started":1}
I20260620 14:52:25.901914 20212 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:52:25.890207 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.2:45613 (local address 127.0.0.1:50730)
0620 14:52:25.892448 (+ 2241us) negotiation.cc:107] Waiting for socket to connect
0620 14:52:25.892473 (+ 25us) client_negotiation.cc:174] Beginning negotiation
0620 14:52:25.892585 (+ 112us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 14:52:25.894159 (+ 1574us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 14:52:25.894164 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 14:52:25.894196 (+ 32us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 14:52:25.894462 (+ 266us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:52:25.894470 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:52:25.896495 (+ 2025us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:52:25.896501 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:52:25.898783 (+ 2282us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:52:25.898798 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:52:25.900974 (+ 2176us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:52:25.900984 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:52:25.901173 (+ 189us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:52:25.901208 (+ 35us) client_negotiation.cc:769] Sending connection context
0620 14:52:25.901387 (+ 179us) client_negotiation.cc:240] Negotiation successful
0620 14:52:25.901525 (+ 138us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":2096,"mutex_wait_us":7,"thread_start_us":1912,"threads_started":1}
I20260620 14:52:25.902248 20213 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:52:25.890710 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50730 (local address 127.0.97.2:45613)
0620 14:52:25.892134 (+ 1424us) server_negotiation.cc:206] Beginning negotiation
0620 14:52:25.892142 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0620 14:52:25.893002 (+ 860us) server_negotiation.cc:407] Connection header received
0620 14:52:25.893232 (+ 230us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:52:25.893253 (+ 21us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:52:25.893677 (+ 424us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:52:25.893930 (+ 253us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 14:52:25.894624 (+ 694us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:52:25.896331 (+ 1707us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:52:25.899000 (+ 2669us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:52:25.900654 (+ 1654us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:52:25.900803 (+ 149us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:52:25.901002 (+ 199us) server_negotiation.cc:1035] Waiting for connection context
0620 14:52:25.901740 (+ 738us) server_negotiation.cc:299] Negotiation successful
0620 14:52:25.901933 (+ 193us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1275,"thread_start_us":1174,"threads_started":1}
I20260620 14:52:25.905476 19862 tablet_service.cc:1467] Processing CreateTablet for tablet f10dc20aabf140259cac2efa9912cf29 (DEFAULT_TABLE table=test-table [id=5b9f752676934966b518416a294ecb0a]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 14:52:25.905495 20130 tablet_service.cc:1467] Processing CreateTablet for tablet f10dc20aabf140259cac2efa9912cf29 (DEFAULT_TABLE table=test-table [id=5b9f752676934966b518416a294ecb0a]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 14:52:25.905720 19996 tablet_service.cc:1467] Processing CreateTablet for tablet f10dc20aabf140259cac2efa9912cf29 (DEFAULT_TABLE table=test-table [id=5b9f752676934966b518416a294ecb0a]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 14:52:25.906942 19862 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f10dc20aabf140259cac2efa9912cf29. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 14:52:25.906980 19996 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f10dc20aabf140259cac2efa9912cf29. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 14:52:25.906981 20130 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f10dc20aabf140259cac2efa9912cf29. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 14:52:25.919770 20214 tablet_bootstrap.cc:492] T f10dc20aabf140259cac2efa9912cf29 P 64e8c290414645e8a44a84e22960df76: Bootstrap starting.
I20260620 14:52:25.924068 20214 tablet_bootstrap.cc:654] T f10dc20aabf140259cac2efa9912cf29 P 64e8c290414645e8a44a84e22960df76: Neither blocks nor log segments found. Creating new log.
I20260620 14:52:25.925300 20214 log.cc:824] T f10dc20aabf140259cac2efa9912cf29 P 64e8c290414645e8a44a84e22960df76: Log is configured to *not* fsync() on all Append() calls
I20260620 14:52:25.926003 20216 tablet_bootstrap.cc:492] T f10dc20aabf140259cac2efa9912cf29 P 33401df5fb2c4a818f615a24ab8c5825: Bootstrap starting.
I20260620 14:52:25.928890 20214 tablet_bootstrap.cc:492] T f10dc20aabf140259cac2efa9912cf29 P 64e8c290414645e8a44a84e22960df76: No bootstrap required, opened a new log
I20260620 14:52:25.929235 20214 ts_tablet_manager.cc:1397] T f10dc20aabf140259cac2efa9912cf29 P 64e8c290414645e8a44a84e22960df76: Time spent bootstrapping tablet: real 0.010s user 0.008s sys 0.000s
I20260620 14:52:25.929956 20215 tablet_bootstrap.cc:492] T f10dc20aabf140259cac2efa9912cf29 P 096323af948c42deb241325687a37eaa: Bootstrap starting.
I20260620 14:52:25.930760 20216 tablet_bootstrap.cc:654] T f10dc20aabf140259cac2efa9912cf29 P 33401df5fb2c4a818f615a24ab8c5825: Neither blocks nor log segments found. Creating new log.
I20260620 14:52:25.932163 20216 log.cc:824] T f10dc20aabf140259cac2efa9912cf29 P 33401df5fb2c4a818f615a24ab8c5825: Log is configured to *not* fsync() on all Append() calls
I20260620 14:52:25.934792 20215 tablet_bootstrap.cc:654] T f10dc20aabf140259cac2efa9912cf29 P 096323af948c42deb241325687a37eaa: Neither blocks nor log segments found. Creating new log.
I20260620 14:52:25.935194 20216 tablet_bootstrap.cc:492] T f10dc20aabf140259cac2efa9912cf29 P 33401df5fb2c4a818f615a24ab8c5825: No bootstrap required, opened a new log
I20260620 14:52:25.935503 20216 ts_tablet_manager.cc:1397] T f10dc20aabf140259cac2efa9912cf29 P 33401df5fb2c4a818f615a24ab8c5825: Time spent bootstrapping tablet: real 0.010s user 0.007s sys 0.000s
I20260620 14:52:25.936197 20215 log.cc:824] T f10dc20aabf140259cac2efa9912cf29 P 096323af948c42deb241325687a37eaa: Log is configured to *not* fsync() on all Append() calls
I20260620 14:52:25.939797 20215 tablet_bootstrap.cc:492] T f10dc20aabf140259cac2efa9912cf29 P 096323af948c42deb241325687a37eaa: No bootstrap required, opened a new log
I20260620 14:52:25.940135 20215 ts_tablet_manager.cc:1397] T f10dc20aabf140259cac2efa9912cf29 P 096323af948c42deb241325687a37eaa: Time spent bootstrapping tablet: real 0.010s user 0.007s sys 0.002s
I20260620 14:52:25.940096 20214 raft_consensus.cc:348] T f10dc20aabf140259cac2efa9912cf29 P 64e8c290414645e8a44a84e22960df76 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "096323af948c42deb241325687a37eaa" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 39597 } } peers { permanent_uuid: "64e8c290414645e8a44a84e22960df76" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 33989 } } peers { permanent_uuid: "33401df5fb2c4a818f615a24ab8c5825" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 45613 } }
I20260620 14:52:25.940621 20214 raft_consensus.cc:374] T f10dc20aabf140259cac2efa9912cf29 P 64e8c290414645e8a44a84e22960df76 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 14:52:25.940738 20214 raft_consensus.cc:729] T f10dc20aabf140259cac2efa9912cf29 P 64e8c290414645e8a44a84e22960df76 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 64e8c290414645e8a44a84e22960df76, State: Initialized, Role: FOLLOWER
I20260620 14:52:25.941475 20214 consensus_queue.cc:260] T f10dc20aabf140259cac2efa9912cf29 P 64e8c290414645e8a44a84e22960df76 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "096323af948c42deb241325687a37eaa" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 39597 } } peers { permanent_uuid: "64e8c290414645e8a44a84e22960df76" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 33989 } } peers { permanent_uuid: "33401df5fb2c4a818f615a24ab8c5825" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 45613 } }
I20260620 14:52:25.944432 20216 raft_consensus.cc:348] T f10dc20aabf140259cac2efa9912cf29 P 33401df5fb2c4a818f615a24ab8c5825 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "096323af948c42deb241325687a37eaa" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 39597 } } peers { permanent_uuid: "64e8c290414645e8a44a84e22960df76" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 33989 } } peers { permanent_uuid: "33401df5fb2c4a818f615a24ab8c5825" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 45613 } }
I20260620 14:52:25.944845 20216 raft_consensus.cc:374] T f10dc20aabf140259cac2efa9912cf29 P 33401df5fb2c4a818f615a24ab8c5825 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 14:52:25.944957 20216 raft_consensus.cc:729] T f10dc20aabf140259cac2efa9912cf29 P 33401df5fb2c4a818f615a24ab8c5825 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 33401df5fb2c4a818f615a24ab8c5825, State: Initialized, Role: FOLLOWER
I20260620 14:52:25.945840 20216 consensus_queue.cc:260] T f10dc20aabf140259cac2efa9912cf29 P 33401df5fb2c4a818f615a24ab8c5825 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "096323af948c42deb241325687a37eaa" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 39597 } } peers { permanent_uuid: "64e8c290414645e8a44a84e22960df76" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 33989 } } peers { permanent_uuid: "33401df5fb2c4a818f615a24ab8c5825" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 45613 } }
I20260620 14:52:25.946523 20214 ts_tablet_manager.cc:1428] T f10dc20aabf140259cac2efa9912cf29 P 64e8c290414645e8a44a84e22960df76: Time spent starting tablet: real 0.017s user 0.016s sys 0.000s
I20260620 14:52:25.946774 20220 raft_consensus.cc:482] T f10dc20aabf140259cac2efa9912cf29 P 64e8c290414645e8a44a84e22960df76 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260620 14:52:25.947095 20220 raft_consensus.cc:504] T f10dc20aabf140259cac2efa9912cf29 P 64e8c290414645e8a44a84e22960df76 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "096323af948c42deb241325687a37eaa" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 39597 } } peers { permanent_uuid: "64e8c290414645e8a44a84e22960df76" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 33989 } } peers { permanent_uuid: "33401df5fb2c4a818f615a24ab8c5825" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 45613 } }
I20260620 14:52:25.949704 20220 leader_election.cc:288] T f10dc20aabf140259cac2efa9912cf29 P 64e8c290414645e8a44a84e22960df76 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 096323af948c42deb241325687a37eaa (127.0.97.3:39597), 33401df5fb2c4a818f615a24ab8c5825 (127.0.97.2:45613)
I20260620 14:52:25.954209 20061 heartbeater.cc:503] Master 127.0.97.62:43229 was elected leader, sending a full tablet report...
I20260620 14:52:25.954069 20215 raft_consensus.cc:348] T f10dc20aabf140259cac2efa9912cf29 P 096323af948c42deb241325687a37eaa [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "096323af948c42deb241325687a37eaa" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 39597 } } peers { permanent_uuid: "64e8c290414645e8a44a84e22960df76" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 33989 } } peers { permanent_uuid: "33401df5fb2c4a818f615a24ab8c5825" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 45613 } }
I20260620 14:52:25.954624 20215 raft_consensus.cc:374] T f10dc20aabf140259cac2efa9912cf29 P 096323af948c42deb241325687a37eaa [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 14:52:25.954741 20215 raft_consensus.cc:729] T f10dc20aabf140259cac2efa9912cf29 P 096323af948c42deb241325687a37eaa [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 096323af948c42deb241325687a37eaa, State: Initialized, Role: FOLLOWER
I20260620 14:52:25.955389 20216 ts_tablet_manager.cc:1428] T f10dc20aabf140259cac2efa9912cf29 P 33401df5fb2c4a818f615a24ab8c5825: Time spent starting tablet: real 0.020s user 0.008s sys 0.004s
I20260620 14:52:25.955452 20215 consensus_queue.cc:260] T f10dc20aabf140259cac2efa9912cf29 P 096323af948c42deb241325687a37eaa [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "096323af948c42deb241325687a37eaa" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 39597 } } peers { permanent_uuid: "64e8c290414645e8a44a84e22960df76" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 33989 } } peers { permanent_uuid: "33401df5fb2c4a818f615a24ab8c5825" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 45613 } }
I20260620 14:52:25.963248 20213 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:52:25.953343 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.1:55659 (local address 127.0.97.2:45613)
0620 14:52:25.953568 (+ 225us) server_negotiation.cc:206] Beginning negotiation
0620 14:52:25.953577 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0620 14:52:25.956657 (+ 3080us) server_negotiation.cc:407] Connection header received
0620 14:52:25.956822 (+ 165us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:52:25.956830 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:52:25.956909 (+ 79us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:52:25.957017 (+ 108us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 14:52:25.957947 (+ 930us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:52:25.958971 (+ 1024us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:52:25.961193 (+ 2222us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:52:25.962232 (+ 1039us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:52:25.962415 (+ 183us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:52:25.962522 (+ 107us) server_negotiation.cc:1035] Waiting for connection context
0620 14:52:25.962828 (+ 306us) server_negotiation.cc:299] Negotiation successful
0620 14:52:25.962960 (+ 132us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":46}
I20260620 14:52:25.963313 20222 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:52:25.951888 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.2:45613 (local address 127.0.97.1:55659)
0620 14:52:25.956507 (+ 4619us) negotiation.cc:107] Waiting for socket to connect
0620 14:52:25.956533 (+ 26us) client_negotiation.cc:174] Beginning negotiation
0620 14:52:25.956692 (+ 159us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 14:52:25.957211 (+ 519us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 14:52:25.957216 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 14:52:25.957249 (+ 33us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 14:52:25.957720 (+ 471us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:52:25.957744 (+ 24us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:52:25.959173 (+ 1429us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:52:25.959178 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:52:25.961013 (+ 1835us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:52:25.961027 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:52:25.962490 (+ 1463us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:52:25.962497 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:52:25.962658 (+ 161us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:52:25.962683 (+ 25us) client_negotiation.cc:769] Sending connection context
0620 14:52:25.962803 (+ 120us) client_negotiation.cc:240] Negotiation successful
0620 14:52:25.962953 (+ 150us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":4437,"thread_start_us":1663,"threads_started":1}
I20260620 14:52:25.964577 20016 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f10dc20aabf140259cac2efa9912cf29" candidate_uuid: "64e8c290414645e8a44a84e22960df76" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "33401df5fb2c4a818f615a24ab8c5825" is_pre_election: true
I20260620 14:52:25.965122 20016 raft_consensus.cc:2454] T f10dc20aabf140259cac2efa9912cf29 P 33401df5fb2c4a818f615a24ab8c5825 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 64e8c290414645e8a44a84e22960df76 in term 0.
I20260620 14:52:25.965829 19818 leader_election.cc:302] T f10dc20aabf140259cac2efa9912cf29 P 64e8c290414645e8a44a84e22960df76 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 33401df5fb2c4a818f615a24ab8c5825, 64e8c290414645e8a44a84e22960df76; no voters:
I20260620 14:52:25.966262 20220 raft_consensus.cc:2781] T f10dc20aabf140259cac2efa9912cf29 P 64e8c290414645e8a44a84e22960df76 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260620 14:52:25.966439 20220 raft_consensus.cc:482] T f10dc20aabf140259cac2efa9912cf29 P 64e8c290414645e8a44a84e22960df76 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260620 14:52:25.966549 20220 raft_consensus.cc:3037] T f10dc20aabf140259cac2efa9912cf29 P 64e8c290414645e8a44a84e22960df76 [term 0 FOLLOWER]: Advancing to term 1
I20260620 14:52:25.967351 20211 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:52:25.951048 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.1:33525 (local address 127.0.97.3:39597)
0620 14:52:25.951324 (+ 276us) server_negotiation.cc:206] Beginning negotiation
0620 14:52:25.951334 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0620 14:52:25.956919 (+ 5585us) server_negotiation.cc:407] Connection header received
0620 14:52:25.957020 (+ 101us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:52:25.957028 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:52:25.957135 (+ 107us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:52:25.957238 (+ 103us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 14:52:25.958500 (+ 1262us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:52:25.960124 (+ 1624us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:52:25.963838 (+ 3714us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:52:25.965745 (+ 1907us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:52:25.965897 (+ 152us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:52:25.966053 (+ 156us) server_negotiation.cc:1035] Waiting for connection context
0620 14:52:25.966883 (+ 830us) server_negotiation.cc:299] Negotiation successful
0620 14:52:25.967061 (+ 178us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":76}
I20260620 14:52:25.967615 20223 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:52:25.950394 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.3:39597 (local address 127.0.97.1:33525)
0620 14:52:25.953897 (+ 3503us) negotiation.cc:107] Waiting for socket to connect
0620 14:52:25.953919 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0620 14:52:25.955877 (+ 1958us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 14:52:25.957777 (+ 1900us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 14:52:25.957789 (+ 12us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 14:52:25.957836 (+ 47us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 14:52:25.958284 (+ 448us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:52:25.958299 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:52:25.960520 (+ 2221us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:52:25.960528 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:52:25.963513 (+ 2985us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:52:25.963535 (+ 22us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:52:25.966342 (+ 2807us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:52:25.966352 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:52:25.966639 (+ 287us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:52:25.966696 (+ 57us) client_negotiation.cc:769] Sending connection context
0620 14:52:25.966844 (+ 148us) client_negotiation.cc:240] Negotiation successful
0620 14:52:25.967071 (+ 227us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":3357,"thread_start_us":6979,"threads_started":1}
I20260620 14:52:25.968753 20150 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f10dc20aabf140259cac2efa9912cf29" candidate_uuid: "64e8c290414645e8a44a84e22960df76" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "096323af948c42deb241325687a37eaa" is_pre_election: true
W20260620 14:52:25.969005 19928 tablet.cc:2369] T f10dc20aabf140259cac2efa9912cf29 P 64e8c290414645e8a44a84e22960df76: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260620 14:52:25.969358 20150 raft_consensus.cc:2454] T f10dc20aabf140259cac2efa9912cf29 P 096323af948c42deb241325687a37eaa [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 64e8c290414645e8a44a84e22960df76 in term 0.
I20260620 14:52:25.969745 20215 ts_tablet_manager.cc:1428] T f10dc20aabf140259cac2efa9912cf29 P 096323af948c42deb241325687a37eaa: Time spent starting tablet: real 0.029s user 0.018s sys 0.009s
I20260620 14:52:25.970018 20220 raft_consensus.cc:504] T f10dc20aabf140259cac2efa9912cf29 P 64e8c290414645e8a44a84e22960df76 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "096323af948c42deb241325687a37eaa" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 39597 } } peers { permanent_uuid: "64e8c290414645e8a44a84e22960df76" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 33989 } } peers { permanent_uuid: "33401df5fb2c4a818f615a24ab8c5825" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 45613 } }
I20260620 14:52:25.970295 20195 heartbeater.cc:503] Master 127.0.97.62:43229 was elected leader, sending a full tablet report...
I20260620 14:52:25.970834 20220 leader_election.cc:288] T f10dc20aabf140259cac2efa9912cf29 P 64e8c290414645e8a44a84e22960df76 [CANDIDATE]: Term 1 election: Requested vote from peers 096323af948c42deb241325687a37eaa (127.0.97.3:39597), 33401df5fb2c4a818f615a24ab8c5825 (127.0.97.2:45613)
I20260620 14:52:25.971323 20150 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f10dc20aabf140259cac2efa9912cf29" candidate_uuid: "64e8c290414645e8a44a84e22960df76" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "096323af948c42deb241325687a37eaa"
I20260620 14:52:25.971340 20016 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f10dc20aabf140259cac2efa9912cf29" candidate_uuid: "64e8c290414645e8a44a84e22960df76" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "33401df5fb2c4a818f615a24ab8c5825"
I20260620 14:52:25.971551 20150 raft_consensus.cc:3037] T f10dc20aabf140259cac2efa9912cf29 P 096323af948c42deb241325687a37eaa [term 0 FOLLOWER]: Advancing to term 1
I20260620 14:52:25.971541 20016 raft_consensus.cc:3037] T f10dc20aabf140259cac2efa9912cf29 P 33401df5fb2c4a818f615a24ab8c5825 [term 0 FOLLOWER]: Advancing to term 1
I20260620 14:52:25.974954 20150 raft_consensus.cc:2454] T f10dc20aabf140259cac2efa9912cf29 P 096323af948c42deb241325687a37eaa [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 64e8c290414645e8a44a84e22960df76 in term 1.
I20260620 14:52:25.974944 20016 raft_consensus.cc:2454] T f10dc20aabf140259cac2efa9912cf29 P 33401df5fb2c4a818f615a24ab8c5825 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 64e8c290414645e8a44a84e22960df76 in term 1.
I20260620 14:52:25.975517 19816 leader_election.cc:302] T f10dc20aabf140259cac2efa9912cf29 P 64e8c290414645e8a44a84e22960df76 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 096323af948c42deb241325687a37eaa, 64e8c290414645e8a44a84e22960df76; no voters:
I20260620 14:52:25.975916 20220 raft_consensus.cc:2781] T f10dc20aabf140259cac2efa9912cf29 P 64e8c290414645e8a44a84e22960df76 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 14:52:25.976428 20220 raft_consensus.cc:686] T f10dc20aabf140259cac2efa9912cf29 P 64e8c290414645e8a44a84e22960df76 [term 1 LEADER]: Becoming Leader. State: Replica: 64e8c290414645e8a44a84e22960df76, State: Running, Role: LEADER
I20260620 14:52:25.976959 20220 consensus_queue.cc:237] T f10dc20aabf140259cac2efa9912cf29 P 64e8c290414645e8a44a84e22960df76 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "096323af948c42deb241325687a37eaa" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 39597 } } peers { permanent_uuid: "64e8c290414645e8a44a84e22960df76" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 33989 } } peers { permanent_uuid: "33401df5fb2c4a818f615a24ab8c5825" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 45613 } }
I20260620 14:52:25.981986 19738 catalog_manager.cc:5495] T f10dc20aabf140259cac2efa9912cf29 P 64e8c290414645e8a44a84e22960df76 reported cstate change: term changed from 0 to 1, leader changed from <none> to 64e8c290414645e8a44a84e22960df76 (127.0.97.1). New cstate: current_term: 1 leader_uuid: "64e8c290414645e8a44a84e22960df76" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "096323af948c42deb241325687a37eaa" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 39597 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "64e8c290414645e8a44a84e22960df76" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 33989 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "33401df5fb2c4a818f615a24ab8c5825" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 45613 } health_report { overall_health: UNKNOWN } } }
I20260620 14:52:26.004662 20210 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:52:25.999146 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34616 (local address 127.0.97.1:33989)
0620 14:52:25.999353 (+ 207us) server_negotiation.cc:206] Beginning negotiation
0620 14:52:25.999358 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 14:52:25.999377 (+ 19us) server_negotiation.cc:407] Connection header received
0620 14:52:25.999528 (+ 151us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:52:25.999531 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:52:25.999678 (+ 147us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:52:25.999815 (+ 137us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 14:52:26.000155 (+ 340us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:52:26.000878 (+ 723us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:52:26.002305 (+ 1427us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:52:26.002703 (+ 398us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:52:26.002800 (+ 97us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:52:26.003384 (+ 584us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 14:52:26.004037 (+ 653us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0620 14:52:26.004159 (+ 122us) server_negotiation.cc:1035] Waiting for connection context
0620 14:52:26.004253 (+ 94us) server_negotiation.cc:299] Negotiation successful
0620 14:52:26.004343 (+ 90us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":54}
I20260620 14:52:26.013883 20150 raft_consensus.cc:1261] T f10dc20aabf140259cac2efa9912cf29 P 096323af948c42deb241325687a37eaa [term 1 FOLLOWER]: Refusing update from remote peer 64e8c290414645e8a44a84e22960df76: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 14:52:26.013833 20016 raft_consensus.cc:1261] T f10dc20aabf140259cac2efa9912cf29 P 33401df5fb2c4a818f615a24ab8c5825 [term 1 FOLLOWER]: Refusing update from remote peer 64e8c290414645e8a44a84e22960df76: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 14:52:26.015199 20225 consensus_queue.cc:1035] T f10dc20aabf140259cac2efa9912cf29 P 64e8c290414645e8a44a84e22960df76 [LEADER]: Connected to new peer: Peer: permanent_uuid: "33401df5fb2c4a818f615a24ab8c5825" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 45613 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20260620 14:52:26.015444 20062 tablet.cc:2369] T f10dc20aabf140259cac2efa9912cf29 P 33401df5fb2c4a818f615a24ab8c5825: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260620 14:52:26.015659 20220 consensus_queue.cc:1035] T f10dc20aabf140259cac2efa9912cf29 P 64e8c290414645e8a44a84e22960df76 [LEADER]: Connected to new peer: Peer: permanent_uuid: "096323af948c42deb241325687a37eaa" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 39597 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 14:52:26.030591 20228 mvcc.cc:205] Tried to move back new op lower bound from 7298937430061084672 to 7298937429925257216. Current Snapshot: MvccSnapshot[applied={T|T < 7298937430061084672}]
I20260620 14:52:26.032877 20230 mvcc.cc:205] Tried to move back new op lower bound from 7298937430061084672 to 7298937429925257216. Current Snapshot: MvccSnapshot[applied={T|T < 7298937430061084672}]
I20260620 14:52:26.048152 19737 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:51414:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 14:52:26.048501 19737 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:51414:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 14:52:26.052381 19737 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 01c9e681e10441b89d215dc4a02f4525: Sending DeleteTablet for 3 replicas of tablet f10dc20aabf140259cac2efa9912cf29
I20260620 14:52:26.053781 19996 tablet_service.cc:1514] Processing DeleteTablet for tablet f10dc20aabf140259cac2efa9912cf29 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 14:52:26 UTC) from {username='kudu', principal='kudu/127.0.97.62@KRBTEST.COM'} at 127.0.0.1:50730
I20260620 14:52:26.054852 20130 tablet_service.cc:1514] Processing DeleteTablet for tablet f10dc20aabf140259cac2efa9912cf29 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 14:52:26 UTC) from {username='kudu', principal='kudu/127.0.97.62@KRBTEST.COM'} at 127.0.0.1:48592
I20260620 14:52:26.055395 19862 tablet_service.cc:1514] Processing DeleteTablet for tablet f10dc20aabf140259cac2efa9912cf29 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 14:52:26 UTC) from {username='kudu', principal='kudu/127.0.97.62@KRBTEST.COM'} at 127.0.0.1:34600
I20260620 14:52:26.056255 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskUci3DU/build/asan/bin/kudu pid 19799
I20260620 14:52:26.064872 20237 tablet_replica.cc:330] T f10dc20aabf140259cac2efa9912cf29 P 33401df5fb2c4a818f615a24ab8c5825: stopping tablet replica
I20260620 14:52:26.064976 20238 tablet_replica.cc:330] T f10dc20aabf140259cac2efa9912cf29 P 096323af948c42deb241325687a37eaa: stopping tablet replica
I20260620 14:52:26.065426 20237 raft_consensus.cc:2229] T f10dc20aabf140259cac2efa9912cf29 P 33401df5fb2c4a818f615a24ab8c5825 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 14:52:26.065886 20238 raft_consensus.cc:2229] T f10dc20aabf140259cac2efa9912cf29 P 096323af948c42deb241325687a37eaa [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 14:52:26.066007 20237 raft_consensus.cc:2258] T f10dc20aabf140259cac2efa9912cf29 P 33401df5fb2c4a818f615a24ab8c5825 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 14:52:26.066524 20238 raft_consensus.cc:2258] T f10dc20aabf140259cac2efa9912cf29 P 096323af948c42deb241325687a37eaa [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 14:52:26.067153 20239 tablet_replica.cc:330] T f10dc20aabf140259cac2efa9912cf29 P 64e8c290414645e8a44a84e22960df76: stopping tablet replica
I20260620 14:52:26.067682 20239 raft_consensus.cc:2229] T f10dc20aabf140259cac2efa9912cf29 P 64e8c290414645e8a44a84e22960df76 [term 1 LEADER]: Raft consensus shutting down.
I20260620 14:52:26.068434 20239 raft_consensus.cc:2258] T f10dc20aabf140259cac2efa9912cf29 P 64e8c290414645e8a44a84e22960df76 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 14:52:26.069525 20238 ts_tablet_manager.cc:1905] T f10dc20aabf140259cac2efa9912cf29 P 096323af948c42deb241325687a37eaa: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 14:52:26.070739 20237 ts_tablet_manager.cc:1905] T f10dc20aabf140259cac2efa9912cf29 P 33401df5fb2c4a818f615a24ab8c5825: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 14:52:26.074519 20239 ts_tablet_manager.cc:1905] T f10dc20aabf140259cac2efa9912cf29 P 64e8c290414645e8a44a84e22960df76: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 14:52:26.076421 20238 ts_tablet_manager.cc:1918] T f10dc20aabf140259cac2efa9912cf29 P 096323af948c42deb241325687a37eaa: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260620 14:52:26.076627 20238 log.cc:1196] T f10dc20aabf140259cac2efa9912cf29 P 096323af948c42deb241325687a37eaa: Deleting WAL directory at /tmp/dist-test-taskUci3DU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781966829091300-388-0/minicluster-data/ts-2/wal/wals/f10dc20aabf140259cac2efa9912cf29
I20260620 14:52:26.077301 20238 ts_tablet_manager.cc:1939] T f10dc20aabf140259cac2efa9912cf29 P 096323af948c42deb241325687a37eaa: Deleting consensus metadata
I20260620 14:52:26.079512 19723 catalog_manager.cc:4841] TS 096323af948c42deb241325687a37eaa (127.0.97.3:39597): tablet f10dc20aabf140259cac2efa9912cf29 (table test-table [id=5b9f752676934966b518416a294ecb0a]) successfully deleted
Jun 20 14:52:26 dist-test-slave-q9j6 krb5kdc[19693](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781967143, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.1@KRBTEST.COM
I20260620 14:52:26.083408 20239 ts_tablet_manager.cc:1918] T f10dc20aabf140259cac2efa9912cf29 P 64e8c290414645e8a44a84e22960df76: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260620 14:52:26.083673 20239 log.cc:1196] T f10dc20aabf140259cac2efa9912cf29 P 64e8c290414645e8a44a84e22960df76: Deleting WAL directory at /tmp/dist-test-taskUci3DU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781966829091300-388-0/minicluster-data/ts-0/wal/wals/f10dc20aabf140259cac2efa9912cf29
I20260620 14:52:26.084729 20237 ts_tablet_manager.cc:1918] T f10dc20aabf140259cac2efa9912cf29 P 33401df5fb2c4a818f615a24ab8c5825: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260620 14:52:26.084988 20237 log.cc:1196] T f10dc20aabf140259cac2efa9912cf29 P 33401df5fb2c4a818f615a24ab8c5825: Deleting WAL directory at /tmp/dist-test-taskUci3DU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781966829091300-388-0/minicluster-data/ts-1/wal/wals/f10dc20aabf140259cac2efa9912cf29
I20260620 14:52:26.085459 20237 ts_tablet_manager.cc:1939] T f10dc20aabf140259cac2efa9912cf29 P 33401df5fb2c4a818f615a24ab8c5825: Deleting consensus metadata
I20260620 14:52:26.087316 20239 ts_tablet_manager.cc:1939] T f10dc20aabf140259cac2efa9912cf29 P 64e8c290414645e8a44a84e22960df76: Deleting consensus metadata
I20260620 14:52:26.087255 19725 catalog_manager.cc:4841] TS 33401df5fb2c4a818f615a24ab8c5825 (127.0.97.2:45613): tablet f10dc20aabf140259cac2efa9912cf29 (table test-table [id=5b9f752676934966b518416a294ecb0a]) successfully deleted
I20260620 14:52:26.089398 19724 catalog_manager.cc:4841] TS 64e8c290414645e8a44a84e22960df76 (127.0.97.1:33989): tablet f10dc20aabf140259cac2efa9912cf29 (table test-table [id=5b9f752676934966b518416a294ecb0a]) successfully deleted
I20260620 14:52:26.091400 20210 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:52:26.058672 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34630 (local address 127.0.97.1:33989)
0620 14:52:26.058943 (+ 271us) server_negotiation.cc:206] Beginning negotiation
0620 14:52:26.058953 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0620 14:52:26.064275 (+ 5322us) server_negotiation.cc:407] Connection header received
0620 14:52:26.065495 (+ 1220us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:52:26.065500 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:52:26.065595 (+ 95us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:52:26.065717 (+ 122us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:52:26.067708 (+ 1991us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:52:26.069004 (+ 1296us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:52:26.073614 (+ 4610us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:52:26.074407 (+ 793us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:52:26.074538 (+ 131us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:52:26.083434 (+ 8896us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 14:52:26.083455 (+ 21us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 14:52:26.083481 (+ 26us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 14:52:26.083543 (+ 62us) server_negotiation.cc:922] calling sasl_server_start()
0620 14:52:26.087946 (+ 4403us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:52:26.088979 (+ 1033us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:52:26.088988 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:52:26.088992 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:52:26.089243 (+ 251us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:52:26.089713 (+ 470us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:52:26.089719 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:52:26.089722 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:52:26.090184 (+ 462us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 14:52:26.090552 (+ 368us) server_negotiation.cc:1035] Waiting for connection context
0620 14:52:26.090733 (+ 181us) server_negotiation.cc:299] Negotiation successful
0620 14:52:26.090986 (+ 253us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":77}
I20260620 14:52:26.092173 19922 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 14:52:26.194888 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskUci3DU/build/asan/bin/kudu with pid 19799
I20260620 14:52:26.214205 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskUci3DU/build/asan/bin/kudu pid 19933
Jun 20 14:52:26 dist-test-slave-q9j6 krb5kdc[19693](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781967143, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.2@KRBTEST.COM
I20260620 14:52:26.230319 20213 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:52:26.215085 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50738 (local address 127.0.97.2:45613)
0620 14:52:26.215291 (+ 206us) server_negotiation.cc:206] Beginning negotiation
0620 14:52:26.215296 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 14:52:26.215318 (+ 22us) server_negotiation.cc:407] Connection header received
0620 14:52:26.215513 (+ 195us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:52:26.215518 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:52:26.215586 (+ 68us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:52:26.215766 (+ 180us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:52:26.217131 (+ 1365us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:52:26.217882 (+ 751us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:52:26.219260 (+ 1378us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:52:26.219755 (+ 495us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:52:26.219906 (+ 151us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:52:26.224407 (+ 4501us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 14:52:26.224423 (+ 16us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 14:52:26.224443 (+ 20us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 14:52:26.224492 (+ 49us) server_negotiation.cc:922] calling sasl_server_start()
0620 14:52:26.227516 (+ 3024us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:52:26.228494 (+ 978us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:52:26.228501 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:52:26.228503 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:52:26.228649 (+ 146us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:52:26.229014 (+ 365us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:52:26.229017 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:52:26.229019 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:52:26.229391 (+ 372us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 14:52:26.229610 (+ 219us) server_negotiation.cc:1035] Waiting for connection context
0620 14:52:26.229880 (+ 270us) server_negotiation.cc:299] Negotiation successful
0620 14:52:26.230057 (+ 177us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":73}
I20260620 14:52:26.230998 20056 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 14:52:26.322491 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskUci3DU/build/asan/bin/kudu with pid 19933
I20260620 14:52:26.339906 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskUci3DU/build/asan/bin/kudu pid 20067
Jun 20 14:52:26 dist-test-slave-q9j6 krb5kdc[19693](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781967143, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.3@KRBTEST.COM
I20260620 14:52:26.356436 20211 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:52:26.340783 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48598 (local address 127.0.97.3:39597)
0620 14:52:26.340979 (+ 196us) server_negotiation.cc:206] Beginning negotiation
0620 14:52:26.340984 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 14:52:26.341003 (+ 19us) server_negotiation.cc:407] Connection header received
0620 14:52:26.341167 (+ 164us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:52:26.341171 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:52:26.341253 (+ 82us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:52:26.341369 (+ 116us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:52:26.342669 (+ 1300us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:52:26.343413 (+ 744us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:52:26.344850 (+ 1437us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:52:26.345240 (+ 390us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:52:26.345354 (+ 114us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:52:26.350036 (+ 4682us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 14:52:26.350058 (+ 22us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 14:52:26.350076 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 14:52:26.350123 (+ 47us) server_negotiation.cc:922] calling sasl_server_start()
0620 14:52:26.353275 (+ 3152us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:52:26.354147 (+ 872us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:52:26.354156 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:52:26.354158 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:52:26.354318 (+ 160us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:52:26.354733 (+ 415us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:52:26.354738 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:52:26.354743 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:52:26.355140 (+ 397us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 14:52:26.355465 (+ 325us) server_negotiation.cc:1035] Waiting for connection context
0620 14:52:26.355829 (+ 364us) server_negotiation.cc:299] Negotiation successful
0620 14:52:26.356069 (+ 240us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":71}
I20260620 14:52:26.357098 20190 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 14:52:26.450428 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskUci3DU/build/asan/bin/kudu with pid 20067
I20260620 14:52:26.468112 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskUci3DU/build/asan/bin/kudu pid 19706
I20260620 14:52:26.469349 19768 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 14:52:26.583186 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskUci3DU/build/asan/bin/kudu with pid 19706
2026-06-20T14:52:26Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/4 (4403 ms)
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/5
2026-06-20T14:52:26Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T14:52:26Z Disabled control of system clock
I20260620 14:52:26.665314 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskUci3DU/build/asan/bin/kudu
/tmp/dist-test-taskUci3DU/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskUci3DU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781966829091300-388-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskUci3DU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781966829091300-388-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskUci3DU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781966829091300-388-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskUci3DU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781966829091300-388-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:45331
--webserver_interface=127.0.97.62
--webserver_port=0
--builtin_ntp_servers=127.0.97.20:34663
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.97.62:45331
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260620 14:52:27.012620 20248 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 14:52:27.012948 20248 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 14:52:27.013110 20248 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 14:52:27.020830 20248 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 14:52:27.020958 20248 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 14:52:27.021034 20248 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 14:52:27.021128 20248 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 14:52:27.021170 20248 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 14:52:27.031292 20248 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:34663
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskUci3DU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781966829091300-388-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskUci3DU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781966829091300-388-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.97.62:45331
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:45331
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskUci3DU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781966829091300-388-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.97.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskUci3DU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781966829091300-388-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 13:41:20 UTC on ccycloud.quasar-vdagoi.root.comops.site
build id 4457462
ASAN enabled
I20260620 14:52:27.032722 20248 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 14:52:27.034679 20248 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 14:52:27.047710 20253 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 14:52:27.047742 20256 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 14:52:27.047684 20254 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 14:52:27.048396 20248 server_base.cc:1034] running on GCE node
I20260620 14:52:27.049321 20248 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 14:52:27.050942 20248 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 14:52:27.052225 20248 hybrid_clock.cc:648] HybridClock initialized: now 1781967147052183 us; error 40 us; skew 500 ppm
I20260620 14:52:27.052659 20248 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260620 14:52:27.055253 20248 webserver.cc:460] Webserver started at http://127.0.97.62:42335/ using document root <none> and password file <none>
I20260620 14:52:27.056115 20248 fs_manager.cc:359] Metadata directory not provided
I20260620 14:52:27.056257 20248 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 14:52:27.056799 20248 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 14:52:27.060168 20248 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskUci3DU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781966829091300-388-0/minicluster-data/master-0/data/instance:
uuid: "937605b6de7d4df4a16f746ca2dd382c"
format_stamp: "Formatted at 2026-06-20 14:52:27 on dist-test-slave-q9j6"
server_key: "1cd0acf9910fb2713967ca50729426c2"
server_key_iv: "614d5eab98d917401c99bab909672835"
server_key_version: "encryptionkey@0"
I20260620 14:52:27.061133 20248 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskUci3DU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781966829091300-388-0/minicluster-data/master-0/wal/instance:
uuid: "937605b6de7d4df4a16f746ca2dd382c"
format_stamp: "Formatted at 2026-06-20 14:52:27 on dist-test-slave-q9j6"
server_key: "1cd0acf9910fb2713967ca50729426c2"
server_key_iv: "614d5eab98d917401c99bab909672835"
server_key_version: "encryptionkey@0"
I20260620 14:52:27.067317 20248 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.006s sys 0.000s
I20260620 14:52:27.071375 20262 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 14:52:27.073078 20248 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260620 14:52:27.073419 20248 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskUci3DU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781966829091300-388-0/minicluster-data/master-0/data,/tmp/dist-test-taskUci3DU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781966829091300-388-0/minicluster-data/master-0/wal
uuid: "937605b6de7d4df4a16f746ca2dd382c"
format_stamp: "Formatted at 2026-06-20 14:52:27 on dist-test-slave-q9j6"
server_key: "1cd0acf9910fb2713967ca50729426c2"
server_key_iv: "614d5eab98d917401c99bab909672835"
server_key_version: "encryptionkey@0"
I20260620 14:52:27.073689 20248 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskUci3DU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781966829091300-388-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskUci3DU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781966829091300-388-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskUci3DU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781966829091300-388-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 14:52:27.095453 20248 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 14:52:27.096453 20248 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 14:52:27.096777 20248 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 14:52:27.117350 20248 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.62:45331
I20260620 14:52:27.117374 20313 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.62:45331 every 8 connection(s)
I20260620 14:52:27.118865 20248 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskUci3DU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781966829091300-388-0/minicluster-data/master-0/data/info.pb
I20260620 14:52:27.121236 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskUci3DU/build/asan/bin/kudu as pid 20248
I20260620 14:52:27.121419 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskUci3DU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781966829091300-388-0/minicluster-data/master-0/wal/instance
I20260620 14:52:27.122001 388 external_mini_cluster.cc:1428] Setting key 36fa86d3bb25985b134de07a58be0ce8
I20260620 14:52:27.123872 20314 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 14:52:27.136750 20317 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:52:27.124231 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60324 (local address 127.0.97.62:45331)
0620 14:52:27.125259 (+ 1028us) server_negotiation.cc:206] Beginning negotiation
0620 14:52:27.125279 (+ 20us) server_negotiation.cc:399] Waiting for connection header
0620 14:52:27.125333 (+ 54us) server_negotiation.cc:407] Connection header received
0620 14:52:27.127004 (+ 1671us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:52:27.127031 (+ 27us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:52:27.127731 (+ 700us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:52:27.128202 (+ 471us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:52:27.128852 (+ 650us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:52:27.130223 (+ 1371us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:52:27.131780 (+ 1557us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:52:27.132535 (+ 755us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:52:27.132758 (+ 223us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:52:27.134064 (+ 1306us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 14:52:27.134073 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 14:52:27.134093 (+ 20us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 14:52:27.134756 (+ 663us) server_negotiation.cc:922] calling sasl_server_start()
0620 14:52:27.134851 (+ 95us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 14:52:27.134877 (+ 26us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 14:52:27.135020 (+ 143us) server_negotiation.cc:1035] Waiting for connection context
0620 14:52:27.135289 (+ 269us) server_negotiation.cc:299] Negotiation successful
0620 14:52:27.135720 (+ 431us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":576,"thread_start_us":344,"threads_started":1}
I20260620 14:52:27.137723 20314 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 937605b6de7d4df4a16f746ca2dd382c: Bootstrap starting.
I20260620 14:52:27.141582 20314 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 937605b6de7d4df4a16f746ca2dd382c: Neither blocks nor log segments found. Creating new log.
I20260620 14:52:27.142789 20314 log.cc:824] T 00000000000000000000000000000000 P 937605b6de7d4df4a16f746ca2dd382c: Log is configured to *not* fsync() on all Append() calls
I20260620 14:52:27.146572 20314 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 937605b6de7d4df4a16f746ca2dd382c: No bootstrap required, opened a new log
I20260620 14:52:27.155514 20314 raft_consensus.cc:348] T 00000000000000000000000000000000 P 937605b6de7d4df4a16f746ca2dd382c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "937605b6de7d4df4a16f746ca2dd382c" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 45331 } }
I20260620 14:52:27.156083 20314 raft_consensus.cc:374] T 00000000000000000000000000000000 P 937605b6de7d4df4a16f746ca2dd382c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 14:52:27.156213 20314 raft_consensus.cc:729] T 00000000000000000000000000000000 P 937605b6de7d4df4a16f746ca2dd382c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 937605b6de7d4df4a16f746ca2dd382c, State: Initialized, Role: FOLLOWER
I20260620 14:52:27.156908 20314 consensus_queue.cc:260] T 00000000000000000000000000000000 P 937605b6de7d4df4a16f746ca2dd382c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "937605b6de7d4df4a16f746ca2dd382c" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 45331 } }
I20260620 14:52:27.157189 20314 raft_consensus.cc:388] T 00000000000000000000000000000000 P 937605b6de7d4df4a16f746ca2dd382c [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 14:52:27.157367 20314 raft_consensus.cc:482] T 00000000000000000000000000000000 P 937605b6de7d4df4a16f746ca2dd382c [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 14:52:27.157565 20314 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 937605b6de7d4df4a16f746ca2dd382c [term 0 FOLLOWER]: Advancing to term 1
I20260620 14:52:27.160424 20314 raft_consensus.cc:504] T 00000000000000000000000000000000 P 937605b6de7d4df4a16f746ca2dd382c [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "937605b6de7d4df4a16f746ca2dd382c" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 45331 } }
I20260620 14:52:27.161103 20314 leader_election.cc:302] T 00000000000000000000000000000000 P 937605b6de7d4df4a16f746ca2dd382c [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 937605b6de7d4df4a16f746ca2dd382c; no voters:
I20260620 14:52:27.161789 20314 leader_election.cc:288] T 00000000000000000000000000000000 P 937605b6de7d4df4a16f746ca2dd382c [CANDIDATE]: Term 1 election: Requested vote from peers
I20260620 14:52:27.161957 20319 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 937605b6de7d4df4a16f746ca2dd382c [term 1 FOLLOWER]: Leader election won for term 1
I20260620 14:52:27.162739 20319 raft_consensus.cc:686] T 00000000000000000000000000000000 P 937605b6de7d4df4a16f746ca2dd382c [term 1 LEADER]: Becoming Leader. State: Replica: 937605b6de7d4df4a16f746ca2dd382c, State: Running, Role: LEADER
I20260620 14:52:27.163338 20319 consensus_queue.cc:237] T 00000000000000000000000000000000 P 937605b6de7d4df4a16f746ca2dd382c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "937605b6de7d4df4a16f746ca2dd382c" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 45331 } }
I20260620 14:52:27.164283 20314 sys_catalog.cc:564] T 00000000000000000000000000000000 P 937605b6de7d4df4a16f746ca2dd382c [sys.catalog]: configured and running, proceeding with master startup.
I20260620 14:52:27.166412 20320 sys_catalog.cc:455] T 00000000000000000000000000000000 P 937605b6de7d4df4a16f746ca2dd382c [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "937605b6de7d4df4a16f746ca2dd382c" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "937605b6de7d4df4a16f746ca2dd382c" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 45331 } } }
I20260620 14:52:27.166414 20321 sys_catalog.cc:455] T 00000000000000000000000000000000 P 937605b6de7d4df4a16f746ca2dd382c [sys.catalog]: SysCatalogTable state changed. Reason: New leader 937605b6de7d4df4a16f746ca2dd382c. Latest consensus state: current_term: 1 leader_uuid: "937605b6de7d4df4a16f746ca2dd382c" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "937605b6de7d4df4a16f746ca2dd382c" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 45331 } } }
I20260620 14:52:27.166931 20320 sys_catalog.cc:458] T 00000000000000000000000000000000 P 937605b6de7d4df4a16f746ca2dd382c [sys.catalog]: This master's current role is: LEADER
I20260620 14:52:27.167021 20321 sys_catalog.cc:458] T 00000000000000000000000000000000 P 937605b6de7d4df4a16f746ca2dd382c [sys.catalog]: This master's current role is: LEADER
I20260620 14:52:27.167929 20324 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 14:52:27.174000 20324 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 14:52:27.184695 20324 catalog_manager.cc:1347] Generated new cluster ID: 36bff2ea4e9e4a3b92aae4ea82ad7e17
I20260620 14:52:27.184849 20324 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 14:52:27.210325 20324 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 14:52:27.212539 20324 catalog_manager.cc:1504] Loading token signing keys...
I20260620 14:52:27.227546 20324 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 937605b6de7d4df4a16f746ca2dd382c: Generated new TSK 0
I20260620 14:52:27.228597 20324 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260620 14:52:27.242064 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskUci3DU/build/asan/bin/kudu
/tmp/dist-test-taskUci3DU/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskUci3DU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781966829091300-388-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskUci3DU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781966829091300-388-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskUci3DU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781966829091300-388-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskUci3DU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781966829091300-388-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.1:0
--local_ip_for_outbound_sockets=127.0.97.1
--webserver_interface=127.0.97.1
--webserver_port=0
--tserver_master_addrs=127.0.97.62:45331
--builtin_ntp_servers=127.0.97.20:34663
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260620 14:52:27.618606 20338 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 14:52:27.619010 20338 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 14:52:27.619177 20338 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 14:52:27.628072 20338 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 14:52:27.628230 20338 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 14:52:27.628417 20338 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.1
I20260620 14:52:27.638298 20338 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:34663
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskUci3DU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781966829091300-388-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskUci3DU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781966829091300-388-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskUci3DU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781966829091300-388-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.97.1
--webserver_port=0
--tserver_master_addrs=127.0.97.62:45331
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.1
--log_dir=/tmp/dist-test-taskUci3DU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781966829091300-388-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 13:41:20 UTC on ccycloud.quasar-vdagoi.root.comops.site
build id 4457462
ASAN enabled
I20260620 14:52:27.640121 20338 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 14:52:27.642076 20338 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 14:52:27.656481 20343 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 14:52:27.656679 20344 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 14:52:27.656648 20346 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 14:52:27.657688 20338 server_base.cc:1034] running on GCE node
I20260620 14:52:27.658316 20338 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 14:52:27.659368 20338 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 14:52:27.660573 20338 hybrid_clock.cc:648] HybridClock initialized: now 1781967147660492 us; error 73 us; skew 500 ppm
I20260620 14:52:27.660940 20338 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260620 14:52:27.668231 20338 webserver.cc:460] Webserver started at http://127.0.97.1:35945/ using document root <none> and password file <none>
I20260620 14:52:27.669088 20338 fs_manager.cc:359] Metadata directory not provided
I20260620 14:52:27.669229 20338 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 14:52:27.669598 20338 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 14:52:27.672839 20338 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskUci3DU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781966829091300-388-0/minicluster-data/ts-0/data/instance:
uuid: "18978e1b9066425cabc3a507e86915d5"
format_stamp: "Formatted at 2026-06-20 14:52:27 on dist-test-slave-q9j6"
server_key: "4aaec2a4555324d1e89c948de2c86d2f"
server_key_iv: "d2d0e6ad83922f94f5a537ca24c5b82a"
server_key_version: "encryptionkey@0"
I20260620 14:52:27.673730 20338 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskUci3DU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781966829091300-388-0/minicluster-data/ts-0/wal/instance:
uuid: "18978e1b9066425cabc3a507e86915d5"
format_stamp: "Formatted at 2026-06-20 14:52:27 on dist-test-slave-q9j6"
server_key: "4aaec2a4555324d1e89c948de2c86d2f"
server_key_iv: "d2d0e6ad83922f94f5a537ca24c5b82a"
server_key_version: "encryptionkey@0"
I20260620 14:52:27.679584 20338 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.007s sys 0.000s
I20260620 14:52:27.683753 20352 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 14:52:27.685431 20338 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260620 14:52:27.685657 20338 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskUci3DU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781966829091300-388-0/minicluster-data/ts-0/data,/tmp/dist-test-taskUci3DU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781966829091300-388-0/minicluster-data/ts-0/wal
uuid: "18978e1b9066425cabc3a507e86915d5"
format_stamp: "Formatted at 2026-06-20 14:52:27 on dist-test-slave-q9j6"
server_key: "4aaec2a4555324d1e89c948de2c86d2f"
server_key_iv: "d2d0e6ad83922f94f5a537ca24c5b82a"
server_key_version: "encryptionkey@0"
I20260620 14:52:27.685940 20338 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskUci3DU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781966829091300-388-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskUci3DU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781966829091300-388-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskUci3DU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781966829091300-388-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 14:52:27.709882 20338 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 14:52:27.710814 20338 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 14:52:27.711122 20338 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 14:52:27.712301 20338 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 14:52:27.713900 20338 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 14:52:27.713992 20338 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 14:52:27.714131 20338 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 14:52:27.714213 20338 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 14:52:27.748171 20338 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.1:36767
I20260620 14:52:27.748263 20464 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.1:36767 every 8 connection(s)
I20260620 14:52:27.749557 20338 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskUci3DU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781966829091300-388-0/minicluster-data/ts-0/data/info.pb
I20260620 14:52:27.758180 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskUci3DU/build/asan/bin/kudu as pid 20338
I20260620 14:52:27.758390 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskUci3DU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781966829091300-388-0/minicluster-data/ts-0/wal/instance
I20260620 14:52:27.758822 388 external_mini_cluster.cc:1428] Setting key 6084e88e7f790efbc2b6bea7c8e24705
I20260620 14:52:27.761813 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskUci3DU/build/asan/bin/kudu
/tmp/dist-test-taskUci3DU/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskUci3DU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781966829091300-388-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskUci3DU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781966829091300-388-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskUci3DU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781966829091300-388-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskUci3DU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781966829091300-388-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.2:0
--local_ip_for_outbound_sockets=127.0.97.2
--webserver_interface=127.0.97.2
--webserver_port=0
--tserver_master_addrs=127.0.97.62:45331
--builtin_ntp_servers=127.0.97.20:34663
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
I20260620 14:52:27.762917 20467 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:52:27.752214 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.1:39131 (local address 127.0.97.62:45331)
0620 14:52:27.752706 (+ 492us) server_negotiation.cc:206] Beginning negotiation
0620 14:52:27.752711 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 14:52:27.753803 (+ 1092us) server_negotiation.cc:407] Connection header received
0620 14:52:27.755704 (+ 1901us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:52:27.755710 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:52:27.755820 (+ 110us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:52:27.755949 (+ 129us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:52:27.757070 (+ 1121us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:52:27.757792 (+ 722us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:52:27.759579 (+ 1787us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:52:27.760183 (+ 604us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:52:27.760320 (+ 137us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:52:27.761847 (+ 1527us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 14:52:27.761854 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 14:52:27.761860 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 14:52:27.761884 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0620 14:52:27.761942 (+ 58us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 14:52:27.761962 (+ 20us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 14:52:27.762063 (+ 101us) server_negotiation.cc:1035] Waiting for connection context
0620 14:52:27.762331 (+ 268us) server_negotiation.cc:299] Negotiation successful
0620 14:52:27.762450 (+ 119us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":391,"thread_start_us":274,"threads_started":1}
I20260620 14:52:27.763860 20468 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:52:27.752703 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:45331 (local address 127.0.97.1:39131)
0620 14:52:27.753634 (+ 931us) negotiation.cc:107] Waiting for socket to connect
0620 14:52:27.753671 (+ 37us) client_negotiation.cc:174] Beginning negotiation
0620 14:52:27.755392 (+ 1721us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 14:52:27.756193 (+ 801us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 14:52:27.756202 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 14:52:27.756304 (+ 102us) client_negotiation.cc:189] Negotiated authn=SASL
0620 14:52:27.756894 (+ 590us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:52:27.756911 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:52:27.757969 (+ 1058us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:52:27.757973 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:52:27.759391 (+ 1418us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:52:27.759404 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:52:27.760407 (+ 1003us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:52:27.760413 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:52:27.760728 (+ 315us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:52:27.761480 (+ 752us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0620 14:52:27.761482 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0620 14:52:27.761531 (+ 49us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0620 14:52:27.761619 (+ 88us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 14:52:27.762110 (+ 491us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 14:52:27.762120 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 14:52:27.762126 (+ 6us) client_negotiation.cc:769] Sending connection context
0620 14:52:27.762281 (+ 155us) client_negotiation.cc:240] Negotiation successful
0620 14:52:27.762615 (+ 334us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":463,"thread_start_us":271,"threads_started":1}
I20260620 14:52:27.765586 20465 heartbeater.cc:347] Connected to a master server at 127.0.97.62:45331
I20260620 14:52:27.765975 20465 heartbeater.cc:464] Registering TS with master...
I20260620 14:52:27.766770 20465 heartbeater.cc:511] Master 127.0.97.62:45331 requested a full tablet report, sending...
I20260620 14:52:27.769220 20279 ts_manager.cc:194] Registered new tserver with Master: 18978e1b9066425cabc3a507e86915d5 (127.0.97.1:36767)
I20260620 14:52:27.770864 20279 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.97.1:39131
W20260620 14:52:28.121888 20469 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 14:52:28.122274 20469 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 14:52:28.122416 20469 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 14:52:28.134238 20469 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 14:52:28.134387 20469 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 14:52:28.134584 20469 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.2
I20260620 14:52:28.146575 20469 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:34663
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskUci3DU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781966829091300-388-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskUci3DU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781966829091300-388-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskUci3DU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781966829091300-388-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.97.2
--webserver_port=0
--tserver_master_addrs=127.0.97.62:45331
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.2
--log_dir=/tmp/dist-test-taskUci3DU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781966829091300-388-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 13:41:20 UTC on ccycloud.quasar-vdagoi.root.comops.site
build id 4457462
ASAN enabled
I20260620 14:52:28.147981 20469 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 14:52:28.149744 20469 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 14:52:28.164081 20475 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 14:52:28.164223 20477 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 14:52:28.164275 20474 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 14:52:28.165095 20469 server_base.cc:1034] running on GCE node
I20260620 14:52:28.165998 20469 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 14:52:28.167131 20469 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 14:52:28.168355 20469 hybrid_clock.cc:648] HybridClock initialized: now 1781967148168304 us; error 50 us; skew 500 ppm
I20260620 14:52:28.168740 20469 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260620 14:52:28.171525 20469 webserver.cc:460] Webserver started at http://127.0.97.2:40459/ using document root <none> and password file <none>
I20260620 14:52:28.172367 20469 fs_manager.cc:359] Metadata directory not provided
I20260620 14:52:28.172500 20469 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 14:52:28.172857 20469 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 14:52:28.176137 20469 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskUci3DU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781966829091300-388-0/minicluster-data/ts-1/data/instance:
uuid: "de124445be1e46b88cb67e1db82518aa"
format_stamp: "Formatted at 2026-06-20 14:52:28 on dist-test-slave-q9j6"
server_key: "45af3f2f80d5056d51576afe4027d7df"
server_key_iv: "1a63b602ad9f69c9b63110d1c8dad773"
server_key_version: "encryptionkey@0"
I20260620 14:52:28.177062 20469 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskUci3DU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781966829091300-388-0/minicluster-data/ts-1/wal/instance:
uuid: "de124445be1e46b88cb67e1db82518aa"
format_stamp: "Formatted at 2026-06-20 14:52:28 on dist-test-slave-q9j6"
server_key: "45af3f2f80d5056d51576afe4027d7df"
server_key_iv: "1a63b602ad9f69c9b63110d1c8dad773"
server_key_version: "encryptionkey@0"
I20260620 14:52:28.182961 20469 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.000s
I20260620 14:52:28.186998 20483 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 14:52:28.188642 20469 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260620 14:52:28.188848 20469 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskUci3DU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781966829091300-388-0/minicluster-data/ts-1/data,/tmp/dist-test-taskUci3DU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781966829091300-388-0/minicluster-data/ts-1/wal
uuid: "de124445be1e46b88cb67e1db82518aa"
format_stamp: "Formatted at 2026-06-20 14:52:28 on dist-test-slave-q9j6"
server_key: "45af3f2f80d5056d51576afe4027d7df"
server_key_iv: "1a63b602ad9f69c9b63110d1c8dad773"
server_key_version: "encryptionkey@0"
I20260620 14:52:28.189105 20469 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskUci3DU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781966829091300-388-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskUci3DU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781966829091300-388-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskUci3DU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781966829091300-388-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 14:52:28.209105 20469 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 14:52:28.210038 20469 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 14:52:28.210353 20469 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 14:52:28.211459 20469 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 14:52:28.213160 20469 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 14:52:28.213294 20469 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 14:52:28.213454 20469 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 14:52:28.213541 20469 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 14:52:28.248888 20469 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.2:41915
I20260620 14:52:28.248944 20595 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.2:41915 every 8 connection(s)
I20260620 14:52:28.250411 20469 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskUci3DU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781966829091300-388-0/minicluster-data/ts-1/data/info.pb
I20260620 14:52:28.260142 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskUci3DU/build/asan/bin/kudu as pid 20469
I20260620 14:52:28.260339 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskUci3DU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781966829091300-388-0/minicluster-data/ts-1/wal/instance
I20260620 14:52:28.260785 388 external_mini_cluster.cc:1428] Setting key 6f851505aaff2f477b7d40d46a0dfdf5
I20260620 14:52:28.263890 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskUci3DU/build/asan/bin/kudu
/tmp/dist-test-taskUci3DU/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskUci3DU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781966829091300-388-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskUci3DU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781966829091300-388-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskUci3DU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781966829091300-388-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskUci3DU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781966829091300-388-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.3:0
--local_ip_for_outbound_sockets=127.0.97.3
--webserver_interface=127.0.97.3
--webserver_port=0
--tserver_master_addrs=127.0.97.62:45331
--builtin_ntp_servers=127.0.97.20:34663
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
I20260620 14:52:28.264204 20467 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:52:28.253323 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.2:42519 (local address 127.0.97.62:45331)
0620 14:52:28.253562 (+ 239us) server_negotiation.cc:206] Beginning negotiation
0620 14:52:28.253569 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0620 14:52:28.255125 (+ 1556us) server_negotiation.cc:407] Connection header received
0620 14:52:28.257060 (+ 1935us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:52:28.257064 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:52:28.257140 (+ 76us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:52:28.257224 (+ 84us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:52:28.258414 (+ 1190us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:52:28.259148 (+ 734us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:52:28.261140 (+ 1992us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:52:28.261674 (+ 534us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:52:28.261786 (+ 112us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:52:28.263311 (+ 1525us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 14:52:28.263316 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 14:52:28.263319 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 14:52:28.263337 (+ 18us) server_negotiation.cc:922] calling sasl_server_start()
0620 14:52:28.263368 (+ 31us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 14:52:28.263380 (+ 12us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 14:52:28.263485 (+ 105us) server_negotiation.cc:1035] Waiting for connection context
0620 14:52:28.263784 (+ 299us) server_negotiation.cc:299] Negotiation successful
0620 14:52:28.263887 (+ 103us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":86}
I20260620 14:52:28.265578 20598 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:52:28.253955 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:45331 (local address 127.0.97.2:42519)
0620 14:52:28.254943 (+ 988us) negotiation.cc:107] Waiting for socket to connect
0620 14:52:28.254982 (+ 39us) client_negotiation.cc:174] Beginning negotiation
0620 14:52:28.256748 (+ 1766us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 14:52:28.257479 (+ 731us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 14:52:28.257488 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 14:52:28.257594 (+ 106us) client_negotiation.cc:189] Negotiated authn=SASL
0620 14:52:28.258212 (+ 618us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:52:28.258230 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:52:28.259300 (+ 1070us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:52:28.259303 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:52:28.260956 (+ 1653us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:52:28.260972 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:52:28.261920 (+ 948us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:52:28.261927 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:52:28.262179 (+ 252us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:52:28.262959 (+ 780us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0620 14:52:28.262961 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0620 14:52:28.263007 (+ 46us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0620 14:52:28.263101 (+ 94us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 14:52:28.263488 (+ 387us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 14:52:28.263502 (+ 14us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 14:52:28.263513 (+ 11us) client_negotiation.cc:769] Sending connection context
0620 14:52:28.263775 (+ 262us) client_negotiation.cc:240] Negotiation successful
0620 14:52:28.264242 (+ 467us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":509,"thread_start_us":305,"threads_started":1}
I20260620 14:52:28.267244 20596 heartbeater.cc:347] Connected to a master server at 127.0.97.62:45331
I20260620 14:52:28.267642 20596 heartbeater.cc:464] Registering TS with master...
I20260620 14:52:28.268662 20596 heartbeater.cc:511] Master 127.0.97.62:45331 requested a full tablet report, sending...
I20260620 14:52:28.270624 20279 ts_manager.cc:194] Registered new tserver with Master: de124445be1e46b88cb67e1db82518aa (127.0.97.2:41915)
I20260620 14:52:28.271963 20279 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.97.2:42519
W20260620 14:52:28.616677 20599 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 14:52:28.617116 20599 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 14:52:28.617280 20599 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 14:52:28.626472 20599 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 14:52:28.626595 20599 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 14:52:28.626706 20599 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.3
I20260620 14:52:28.636457 20599 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:34663
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskUci3DU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781966829091300-388-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskUci3DU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781966829091300-388-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskUci3DU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781966829091300-388-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.97.3
--webserver_port=0
--tserver_master_addrs=127.0.97.62:45331
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.3
--log_dir=/tmp/dist-test-taskUci3DU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781966829091300-388-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 13:41:20 UTC on ccycloud.quasar-vdagoi.root.comops.site
build id 4457462
ASAN enabled
I20260620 14:52:28.637831 20599 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 14:52:28.639751 20599 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 14:52:28.654103 20604 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 14:52:28.654067 20607 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 14:52:28.654031 20605 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 14:52:28.654613 20599 server_base.cc:1034] running on GCE node
I20260620 14:52:28.655649 20599 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 14:52:28.656723 20599 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 14:52:28.657945 20599 hybrid_clock.cc:648] HybridClock initialized: now 1781967148657883 us; error 50 us; skew 500 ppm
I20260620 14:52:28.658262 20599 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260620 14:52:28.660971 20599 webserver.cc:460] Webserver started at http://127.0.97.3:37675/ using document root <none> and password file <none>
I20260620 14:52:28.661818 20599 fs_manager.cc:359] Metadata directory not provided
I20260620 14:52:28.661921 20599 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 14:52:28.662204 20599 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 14:52:28.665642 20599 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskUci3DU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781966829091300-388-0/minicluster-data/ts-2/data/instance:
uuid: "8de497f6aea743a1862bf9c6a91b4a2c"
format_stamp: "Formatted at 2026-06-20 14:52:28 on dist-test-slave-q9j6"
server_key: "bdb602d966adc41f45e6b74204832e12"
server_key_iv: "89a943659ad83d8e9ad7973f3871bfff"
server_key_version: "encryptionkey@0"
I20260620 14:52:28.666577 20599 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskUci3DU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781966829091300-388-0/minicluster-data/ts-2/wal/instance:
uuid: "8de497f6aea743a1862bf9c6a91b4a2c"
format_stamp: "Formatted at 2026-06-20 14:52:28 on dist-test-slave-q9j6"
server_key: "bdb602d966adc41f45e6b74204832e12"
server_key_iv: "89a943659ad83d8e9ad7973f3871bfff"
server_key_version: "encryptionkey@0"
I20260620 14:52:28.672674 20599 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.006s sys 0.001s
I20260620 14:52:28.676695 20613 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 14:52:28.678488 20599 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.004s sys 0.000s
I20260620 14:52:28.678699 20599 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskUci3DU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781966829091300-388-0/minicluster-data/ts-2/data,/tmp/dist-test-taskUci3DU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781966829091300-388-0/minicluster-data/ts-2/wal
uuid: "8de497f6aea743a1862bf9c6a91b4a2c"
format_stamp: "Formatted at 2026-06-20 14:52:28 on dist-test-slave-q9j6"
server_key: "bdb602d966adc41f45e6b74204832e12"
server_key_iv: "89a943659ad83d8e9ad7973f3871bfff"
server_key_version: "encryptionkey@0"
I20260620 14:52:28.678957 20599 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskUci3DU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781966829091300-388-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskUci3DU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781966829091300-388-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskUci3DU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781966829091300-388-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 14:52:28.725502 20599 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 14:52:28.726483 20599 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 14:52:28.726809 20599 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 14:52:28.727917 20599 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 14:52:28.729579 20599 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 14:52:28.729713 20599 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 14:52:28.729863 20599 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 14:52:28.729956 20599 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 14:52:28.765897 20599 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.3:44141
I20260620 14:52:28.765941 20725 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.3:44141 every 8 connection(s)
I20260620 14:52:28.767323 20599 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskUci3DU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781966829091300-388-0/minicluster-data/ts-2/data/info.pb
I20260620 14:52:28.774113 20465 heartbeater.cc:503] Master 127.0.97.62:45331 was elected leader, sending a full tablet report...
I20260620 14:52:28.774194 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskUci3DU/build/asan/bin/kudu as pid 20599
I20260620 14:52:28.774381 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskUci3DU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781966829091300-388-0/minicluster-data/ts-2/wal/instance
I20260620 14:52:28.774945 388 external_mini_cluster.cc:1428] Setting key 979c28f34c87ee356fcc9d682ea90438
I20260620 14:52:28.781796 20728 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:52:28.770186 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.3:32857 (local address 127.0.97.62:45331)
0620 14:52:28.770678 (+ 492us) server_negotiation.cc:206] Beginning negotiation
0620 14:52:28.770683 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 14:52:28.772101 (+ 1418us) server_negotiation.cc:407] Connection header received
0620 14:52:28.774106 (+ 2005us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:52:28.774111 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:52:28.774259 (+ 148us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:52:28.774381 (+ 122us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:52:28.776093 (+ 1712us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:52:28.777068 (+ 975us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:52:28.779113 (+ 2045us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:52:28.779512 (+ 399us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:52:28.779671 (+ 159us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:52:28.781027 (+ 1356us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 14:52:28.781032 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 14:52:28.781035 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 14:52:28.781054 (+ 19us) server_negotiation.cc:922] calling sasl_server_start()
0620 14:52:28.781103 (+ 49us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 14:52:28.781116 (+ 13us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 14:52:28.781215 (+ 99us) server_negotiation.cc:1035] Waiting for connection context
0620 14:52:28.781430 (+ 215us) server_negotiation.cc:299] Negotiation successful
0620 14:52:28.781515 (+ 85us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":378,"thread_start_us":264,"threads_started":1}
I20260620 14:52:28.782891 20729 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:52:28.770896 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:45331 (local address 127.0.97.3:32857)
0620 14:52:28.771919 (+ 1023us) negotiation.cc:107] Waiting for socket to connect
0620 14:52:28.771969 (+ 50us) client_negotiation.cc:174] Beginning negotiation
0620 14:52:28.773711 (+ 1742us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 14:52:28.774785 (+ 1074us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 14:52:28.774797 (+ 12us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 14:52:28.774933 (+ 136us) client_negotiation.cc:189] Negotiated authn=SASL
0620 14:52:28.775849 (+ 916us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:52:28.775897 (+ 48us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:52:28.777271 (+ 1374us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:52:28.777280 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:52:28.778942 (+ 1662us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:52:28.778956 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:52:28.779767 (+ 811us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:52:28.779772 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:52:28.779984 (+ 212us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:52:28.780671 (+ 687us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0620 14:52:28.780673 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0620 14:52:28.780720 (+ 47us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0620 14:52:28.780809 (+ 89us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 14:52:28.781228 (+ 419us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 14:52:28.781235 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 14:52:28.781241 (+ 6us) client_negotiation.cc:769] Sending connection context
0620 14:52:28.781386 (+ 145us) client_negotiation.cc:240] Negotiation successful
0620 14:52:28.781714 (+ 328us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":455,"thread_start_us":277,"threads_started":1}
I20260620 14:52:28.784600 20726 heartbeater.cc:347] Connected to a master server at 127.0.97.62:45331
I20260620 14:52:28.784932 20726 heartbeater.cc:464] Registering TS with master...
I20260620 14:52:28.785773 20726 heartbeater.cc:511] Master 127.0.97.62:45331 requested a full tablet report, sending...
I20260620 14:52:28.787381 20279 ts_manager.cc:194] Registered new tserver with Master: 8de497f6aea743a1862bf9c6a91b4a2c (127.0.97.3:44141)
I20260620 14:52:28.788280 20279 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.97.3:32857
I20260620 14:52:28.790925 388 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260620 14:52:28.801261 20728 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:52:28.795471 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60334 (local address 127.0.97.62:45331)
0620 14:52:28.795735 (+ 264us) server_negotiation.cc:206] Beginning negotiation
0620 14:52:28.795744 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0620 14:52:28.796089 (+ 345us) server_negotiation.cc:407] Connection header received
0620 14:52:28.796291 (+ 202us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:52:28.796296 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:52:28.796365 (+ 69us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:52:28.796455 (+ 90us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:52:28.796914 (+ 459us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:52:28.797633 (+ 719us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:52:28.798953 (+ 1320us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:52:28.799362 (+ 409us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:52:28.799464 (+ 102us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:52:28.800537 (+ 1073us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 14:52:28.800541 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 14:52:28.800545 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 14:52:28.800564 (+ 19us) server_negotiation.cc:922] calling sasl_server_start()
0620 14:52:28.800597 (+ 33us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 14:52:28.800609 (+ 12us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 14:52:28.800687 (+ 78us) server_negotiation.cc:1035] Waiting for connection context
0620 14:52:28.800801 (+ 114us) server_negotiation.cc:299] Negotiation successful
0620 14:52:28.800886 (+ 85us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":71}
I20260620 14:52:28.805404 20279 catalog_manager.cc:1908] Servicing CreateTable request from {username='slave'} at 127.0.0.1:60334:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260620 14:52:28.812538 20279 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260620 14:52:28.844861 20740 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:52:28.825904 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:36767 (local address 127.0.0.1:45996)
0620 14:52:28.832640 (+ 6736us) negotiation.cc:107] Waiting for socket to connect
0620 14:52:28.832662 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0620 14:52:28.832942 (+ 280us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 14:52:28.834084 (+ 1142us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 14:52:28.834091 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 14:52:28.834130 (+ 39us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 14:52:28.834648 (+ 518us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:52:28.834667 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:52:28.837488 (+ 2821us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:52:28.837497 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:52:28.840725 (+ 3228us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:52:28.840741 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:52:28.843070 (+ 2329us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:52:28.843076 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:52:28.843921 (+ 845us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:52:28.843961 (+ 40us) client_negotiation.cc:769] Sending connection context
0620 14:52:28.844115 (+ 154us) client_negotiation.cc:240] Negotiation successful
0620 14:52:28.844399 (+ 284us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":6586,"mutex_wait_us":227,"thread_start_us":6415,"threads_started":1}
I20260620 14:52:28.844862 20741 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:52:28.826038 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36816 (local address 127.0.97.3:44141)
0620 14:52:28.832146 (+ 6108us) server_negotiation.cc:206] Beginning negotiation
0620 14:52:28.832156 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0620 14:52:28.832195 (+ 39us) server_negotiation.cc:407] Connection header received
0620 14:52:28.832318 (+ 123us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:52:28.832328 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:52:28.832645 (+ 317us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:52:28.833470 (+ 825us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 14:52:28.833767 (+ 297us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:52:28.835969 (+ 2202us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:52:28.841117 (+ 5148us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:52:28.842642 (+ 1525us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:52:28.842783 (+ 141us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:52:28.843052 (+ 269us) server_negotiation.cc:1035] Waiting for connection context
0620 14:52:28.844326 (+ 1274us) server_negotiation.cc:299] Negotiation successful
0620 14:52:28.844519 (+ 193us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":5917,"thread_start_us":5788,"threads_started":1}
I20260620 14:52:28.845820 20742 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:52:28.826267 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45996 (local address 127.0.97.1:36767)
0620 14:52:28.826942 (+ 675us) server_negotiation.cc:206] Beginning negotiation
0620 14:52:28.826949 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0620 14:52:28.832827 (+ 5878us) server_negotiation.cc:407] Connection header received
0620 14:52:28.833160 (+ 333us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:52:28.833169 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:52:28.833500 (+ 331us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:52:28.833754 (+ 254us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 14:52:28.834909 (+ 1155us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:52:28.837258 (+ 2349us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:52:28.840957 (+ 3699us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:52:28.842493 (+ 1536us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:52:28.842629 (+ 136us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:52:28.842937 (+ 308us) server_negotiation.cc:1035] Waiting for connection context
0620 14:52:28.845188 (+ 2251us) server_negotiation.cc:299] Negotiation successful
0620 14:52:28.845410 (+ 222us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":497,"thread_start_us":356,"threads_started":1}
I20260620 14:52:28.846661 20739 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:52:28.825640 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.3:44141 (local address 127.0.0.1:36816)
0620 14:52:28.828172 (+ 2532us) negotiation.cc:107] Waiting for socket to connect
0620 14:52:28.828214 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0620 14:52:28.828365 (+ 151us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 14:52:28.832917 (+ 4552us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 14:52:28.832926 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 14:52:28.832958 (+ 32us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 14:52:28.833370 (+ 412us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:52:28.833400 (+ 30us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:52:28.836199 (+ 2799us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:52:28.836209 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:52:28.840854 (+ 4645us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:52:28.840871 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:52:28.843365 (+ 2494us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:52:28.843374 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:52:28.843810 (+ 436us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:52:28.843967 (+ 157us) client_negotiation.cc:769] Sending connection context
0620 14:52:28.844114 (+ 147us) client_negotiation.cc:240] Negotiation successful
0620 14:52:28.846209 (+ 2095us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":2255,"mutex_wait_us":1926,"thread_start_us":2116,"threads_started":1}
I20260620 14:52:28.849746 20744 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:52:28.833680 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.2:41915 (local address 127.0.0.1:42874)
0620 14:52:28.840088 (+ 6408us) negotiation.cc:107] Waiting for socket to connect
0620 14:52:28.840114 (+ 26us) client_negotiation.cc:174] Beginning negotiation
0620 14:52:28.840274 (+ 160us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 14:52:28.841043 (+ 769us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 14:52:28.841049 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 14:52:28.841083 (+ 34us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 14:52:28.841483 (+ 400us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:52:28.841495 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:52:28.843362 (+ 1867us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:52:28.843367 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:52:28.846572 (+ 3205us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:52:28.846599 (+ 27us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:52:28.848943 (+ 2344us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:52:28.848951 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:52:28.849132 (+ 181us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:52:28.849168 (+ 36us) client_negotiation.cc:769] Sending connection context
0620 14:52:28.849276 (+ 108us) client_negotiation.cc:240] Negotiation successful
0620 14:52:28.849433 (+ 157us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":6269,"thread_start_us":6144,"threads_started":1}
I20260620 14:52:28.850208 20743 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:52:28.833041 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42874 (local address 127.0.97.2:41915)
0620 14:52:28.839797 (+ 6756us) server_negotiation.cc:206] Beginning negotiation
0620 14:52:28.839807 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0620 14:52:28.840218 (+ 411us) server_negotiation.cc:407] Connection header received
0620 14:52:28.840481 (+ 263us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:52:28.840490 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:52:28.840796 (+ 306us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:52:28.841059 (+ 263us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 14:52:28.841680 (+ 621us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:52:28.843217 (+ 1537us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:52:28.846845 (+ 3628us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:52:28.848388 (+ 1543us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:52:28.848531 (+ 143us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:52:28.848797 (+ 266us) server_negotiation.cc:1035] Waiting for connection context
0620 14:52:28.849353 (+ 556us) server_negotiation.cc:299] Negotiation successful
0620 14:52:28.849528 (+ 175us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":6529,"thread_start_us":6398,"threads_started":1}
I20260620 14:52:28.850358 20400 tablet_service.cc:1467] Processing CreateTablet for tablet 710f5da9fa394eb5affbac105d6f6ae1 (DEFAULT_TABLE table=test-table [id=da48a8822e614e3b88479de387dfb875]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 14:52:28.850560 20661 tablet_service.cc:1467] Processing CreateTablet for tablet 710f5da9fa394eb5affbac105d6f6ae1 (DEFAULT_TABLE table=test-table [id=da48a8822e614e3b88479de387dfb875]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 14:52:28.852102 20400 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 710f5da9fa394eb5affbac105d6f6ae1. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 14:52:28.852128 20661 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 710f5da9fa394eb5affbac105d6f6ae1. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 14:52:28.854372 20531 tablet_service.cc:1467] Processing CreateTablet for tablet 710f5da9fa394eb5affbac105d6f6ae1 (DEFAULT_TABLE table=test-table [id=da48a8822e614e3b88479de387dfb875]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 14:52:28.855967 20531 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 710f5da9fa394eb5affbac105d6f6ae1. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 14:52:28.865900 20745 tablet_bootstrap.cc:492] T 710f5da9fa394eb5affbac105d6f6ae1 P 8de497f6aea743a1862bf9c6a91b4a2c: Bootstrap starting.
I20260620 14:52:28.868942 20746 tablet_bootstrap.cc:492] T 710f5da9fa394eb5affbac105d6f6ae1 P 18978e1b9066425cabc3a507e86915d5: Bootstrap starting.
I20260620 14:52:28.871256 20745 tablet_bootstrap.cc:654] T 710f5da9fa394eb5affbac105d6f6ae1 P 8de497f6aea743a1862bf9c6a91b4a2c: Neither blocks nor log segments found. Creating new log.
I20260620 14:52:28.871800 20747 tablet_bootstrap.cc:492] T 710f5da9fa394eb5affbac105d6f6ae1 P de124445be1e46b88cb67e1db82518aa: Bootstrap starting.
I20260620 14:52:28.872691 20745 log.cc:824] T 710f5da9fa394eb5affbac105d6f6ae1 P 8de497f6aea743a1862bf9c6a91b4a2c: Log is configured to *not* fsync() on all Append() calls
I20260620 14:52:28.872853 20746 tablet_bootstrap.cc:654] T 710f5da9fa394eb5affbac105d6f6ae1 P 18978e1b9066425cabc3a507e86915d5: Neither blocks nor log segments found. Creating new log.
I20260620 14:52:28.874212 20746 log.cc:824] T 710f5da9fa394eb5affbac105d6f6ae1 P 18978e1b9066425cabc3a507e86915d5: Log is configured to *not* fsync() on all Append() calls
I20260620 14:52:28.875774 20747 tablet_bootstrap.cc:654] T 710f5da9fa394eb5affbac105d6f6ae1 P de124445be1e46b88cb67e1db82518aa: Neither blocks nor log segments found. Creating new log.
I20260620 14:52:28.876175 20745 tablet_bootstrap.cc:492] T 710f5da9fa394eb5affbac105d6f6ae1 P 8de497f6aea743a1862bf9c6a91b4a2c: No bootstrap required, opened a new log
I20260620 14:52:28.876487 20745 ts_tablet_manager.cc:1397] T 710f5da9fa394eb5affbac105d6f6ae1 P 8de497f6aea743a1862bf9c6a91b4a2c: Time spent bootstrapping tablet: real 0.011s user 0.005s sys 0.003s
I20260620 14:52:28.876863 20747 log.cc:824] T 710f5da9fa394eb5affbac105d6f6ae1 P de124445be1e46b88cb67e1db82518aa: Log is configured to *not* fsync() on all Append() calls
I20260620 14:52:28.878333 20746 tablet_bootstrap.cc:492] T 710f5da9fa394eb5affbac105d6f6ae1 P 18978e1b9066425cabc3a507e86915d5: No bootstrap required, opened a new log
I20260620 14:52:28.878659 20746 ts_tablet_manager.cc:1397] T 710f5da9fa394eb5affbac105d6f6ae1 P 18978e1b9066425cabc3a507e86915d5: Time spent bootstrapping tablet: real 0.010s user 0.008s sys 0.000s
I20260620 14:52:28.880311 20747 tablet_bootstrap.cc:492] T 710f5da9fa394eb5affbac105d6f6ae1 P de124445be1e46b88cb67e1db82518aa: No bootstrap required, opened a new log
I20260620 14:52:28.880615 20747 ts_tablet_manager.cc:1397] T 710f5da9fa394eb5affbac105d6f6ae1 P de124445be1e46b88cb67e1db82518aa: Time spent bootstrapping tablet: real 0.009s user 0.004s sys 0.003s
I20260620 14:52:28.888341 20747 raft_consensus.cc:348] T 710f5da9fa394eb5affbac105d6f6ae1 P de124445be1e46b88cb67e1db82518aa [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8de497f6aea743a1862bf9c6a91b4a2c" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 44141 } } peers { permanent_uuid: "18978e1b9066425cabc3a507e86915d5" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 36767 } } peers { permanent_uuid: "de124445be1e46b88cb67e1db82518aa" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 41915 } }
I20260620 14:52:28.888815 20747 raft_consensus.cc:374] T 710f5da9fa394eb5affbac105d6f6ae1 P de124445be1e46b88cb67e1db82518aa [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 14:52:28.888928 20747 raft_consensus.cc:729] T 710f5da9fa394eb5affbac105d6f6ae1 P de124445be1e46b88cb67e1db82518aa [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: de124445be1e46b88cb67e1db82518aa, State: Initialized, Role: FOLLOWER
I20260620 14:52:28.888857 20745 raft_consensus.cc:348] T 710f5da9fa394eb5affbac105d6f6ae1 P 8de497f6aea743a1862bf9c6a91b4a2c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8de497f6aea743a1862bf9c6a91b4a2c" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 44141 } } peers { permanent_uuid: "18978e1b9066425cabc3a507e86915d5" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 36767 } } peers { permanent_uuid: "de124445be1e46b88cb67e1db82518aa" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 41915 } }
I20260620 14:52:28.889429 20745 raft_consensus.cc:374] T 710f5da9fa394eb5affbac105d6f6ae1 P 8de497f6aea743a1862bf9c6a91b4a2c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 14:52:28.889549 20745 raft_consensus.cc:729] T 710f5da9fa394eb5affbac105d6f6ae1 P 8de497f6aea743a1862bf9c6a91b4a2c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8de497f6aea743a1862bf9c6a91b4a2c, State: Initialized, Role: FOLLOWER
I20260620 14:52:28.889779 20747 consensus_queue.cc:260] T 710f5da9fa394eb5affbac105d6f6ae1 P de124445be1e46b88cb67e1db82518aa [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8de497f6aea743a1862bf9c6a91b4a2c" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 44141 } } peers { permanent_uuid: "18978e1b9066425cabc3a507e86915d5" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 36767 } } peers { permanent_uuid: "de124445be1e46b88cb67e1db82518aa" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 41915 } }
I20260620 14:52:28.889921 20746 raft_consensus.cc:348] T 710f5da9fa394eb5affbac105d6f6ae1 P 18978e1b9066425cabc3a507e86915d5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8de497f6aea743a1862bf9c6a91b4a2c" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 44141 } } peers { permanent_uuid: "18978e1b9066425cabc3a507e86915d5" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 36767 } } peers { permanent_uuid: "de124445be1e46b88cb67e1db82518aa" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 41915 } }
I20260620 14:52:28.890372 20746 raft_consensus.cc:374] T 710f5da9fa394eb5affbac105d6f6ae1 P 18978e1b9066425cabc3a507e86915d5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 14:52:28.890481 20746 raft_consensus.cc:729] T 710f5da9fa394eb5affbac105d6f6ae1 P 18978e1b9066425cabc3a507e86915d5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 18978e1b9066425cabc3a507e86915d5, State: Initialized, Role: FOLLOWER
I20260620 14:52:28.890251 20745 consensus_queue.cc:260] T 710f5da9fa394eb5affbac105d6f6ae1 P 8de497f6aea743a1862bf9c6a91b4a2c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8de497f6aea743a1862bf9c6a91b4a2c" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 44141 } } peers { permanent_uuid: "18978e1b9066425cabc3a507e86915d5" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 36767 } } peers { permanent_uuid: "de124445be1e46b88cb67e1db82518aa" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 41915 } }
I20260620 14:52:28.891181 20746 consensus_queue.cc:260] T 710f5da9fa394eb5affbac105d6f6ae1 P 18978e1b9066425cabc3a507e86915d5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8de497f6aea743a1862bf9c6a91b4a2c" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 44141 } } peers { permanent_uuid: "18978e1b9066425cabc3a507e86915d5" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 36767 } } peers { permanent_uuid: "de124445be1e46b88cb67e1db82518aa" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 41915 } }
I20260620 14:52:28.891640 20596 heartbeater.cc:503] Master 127.0.97.62:45331 was elected leader, sending a full tablet report...
I20260620 14:52:28.892239 20747 ts_tablet_manager.cc:1428] T 710f5da9fa394eb5affbac105d6f6ae1 P de124445be1e46b88cb67e1db82518aa: Time spent starting tablet: real 0.011s user 0.011s sys 0.001s
I20260620 14:52:28.895138 20745 ts_tablet_manager.cc:1428] T 710f5da9fa394eb5affbac105d6f6ae1 P 8de497f6aea743a1862bf9c6a91b4a2c: Time spent starting tablet: real 0.019s user 0.013s sys 0.001s
I20260620 14:52:28.895910 20726 heartbeater.cc:503] Master 127.0.97.62:45331 was elected leader, sending a full tablet report...
I20260620 14:52:28.897008 20746 ts_tablet_manager.cc:1428] T 710f5da9fa394eb5affbac105d6f6ae1 P 18978e1b9066425cabc3a507e86915d5: Time spent starting tablet: real 0.018s user 0.012s sys 0.005s
I20260620 14:52:28.937295 20751 raft_consensus.cc:482] T 710f5da9fa394eb5affbac105d6f6ae1 P de124445be1e46b88cb67e1db82518aa [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260620 14:52:28.937688 20751 raft_consensus.cc:504] T 710f5da9fa394eb5affbac105d6f6ae1 P de124445be1e46b88cb67e1db82518aa [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8de497f6aea743a1862bf9c6a91b4a2c" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 44141 } } peers { permanent_uuid: "18978e1b9066425cabc3a507e86915d5" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 36767 } } peers { permanent_uuid: "de124445be1e46b88cb67e1db82518aa" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 41915 } }
I20260620 14:52:28.939451 20751 leader_election.cc:288] T 710f5da9fa394eb5affbac105d6f6ae1 P de124445be1e46b88cb67e1db82518aa [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 8de497f6aea743a1862bf9c6a91b4a2c (127.0.97.3:44141), 18978e1b9066425cabc3a507e86915d5 (127.0.97.1:36767)
I20260620 14:52:28.948882 20742 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:52:28.939945 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.2:55401 (local address 127.0.97.1:36767)
0620 14:52:28.940169 (+ 224us) server_negotiation.cc:206] Beginning negotiation
0620 14:52:28.940177 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0620 14:52:28.941091 (+ 914us) server_negotiation.cc:407] Connection header received
0620 14:52:28.941232 (+ 141us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:52:28.941236 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:52:28.941329 (+ 93us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:52:28.941431 (+ 102us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 14:52:28.942182 (+ 751us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:52:28.943287 (+ 1105us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:52:28.946416 (+ 3129us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:52:28.947711 (+ 1295us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:52:28.947850 (+ 139us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:52:28.947959 (+ 109us) server_negotiation.cc:1035] Waiting for connection context
0620 14:52:28.948486 (+ 527us) server_negotiation.cc:299] Negotiation successful
0620 14:52:28.948617 (+ 131us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":65}
I20260620 14:52:28.948896 20741 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:52:28.940304 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.2:52439 (local address 127.0.97.3:44141)
0620 14:52:28.940493 (+ 189us) server_negotiation.cc:206] Beginning negotiation
0620 14:52:28.940500 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0620 14:52:28.940887 (+ 387us) server_negotiation.cc:407] Connection header received
0620 14:52:28.941193 (+ 306us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:52:28.941199 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:52:28.941285 (+ 86us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:52:28.941391 (+ 106us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 14:52:28.942215 (+ 824us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:52:28.943303 (+ 1088us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:52:28.946416 (+ 3113us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:52:28.947718 (+ 1302us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:52:28.947850 (+ 132us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:52:28.947959 (+ 109us) server_negotiation.cc:1035] Waiting for connection context
0620 14:52:28.948486 (+ 527us) server_negotiation.cc:299] Negotiation successful
0620 14:52:28.948617 (+ 131us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":74}
I20260620 14:52:28.949052 20754 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:52:28.939801 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.3:44141 (local address 127.0.97.2:52439)
0620 14:52:28.940735 (+ 934us) negotiation.cc:107] Waiting for socket to connect
0620 14:52:28.940757 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0620 14:52:28.940941 (+ 184us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 14:52:28.941448 (+ 507us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 14:52:28.941453 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 14:52:28.941544 (+ 91us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 14:52:28.942075 (+ 531us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:52:28.942088 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:52:28.943445 (+ 1357us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:52:28.943449 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:52:28.946144 (+ 2695us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:52:28.946163 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:52:28.947953 (+ 1790us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:52:28.947960 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:52:28.948238 (+ 278us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:52:28.948336 (+ 98us) client_negotiation.cc:769] Sending connection context
0620 14:52:28.948439 (+ 103us) client_negotiation.cc:240] Negotiation successful
0620 14:52:28.948635 (+ 196us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":740,"mutex_wait_us":213,"spinlock_wait_cycles":24448,"thread_start_us":550,"threads_started":1}
I20260620 14:52:28.949457 20755 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:52:28.939841 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:36767 (local address 127.0.97.2:55401)
0620 14:52:28.940973 (+ 1132us) negotiation.cc:107] Waiting for socket to connect
0620 14:52:28.940990 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0620 14:52:28.941104 (+ 114us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 14:52:28.941448 (+ 344us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 14:52:28.941453 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 14:52:28.941482 (+ 29us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 14:52:28.941965 (+ 483us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:52:28.941980 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:52:28.943443 (+ 1463us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:52:28.943448 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:52:28.946150 (+ 2702us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:52:28.946163 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:52:28.947953 (+ 1790us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:52:28.947960 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:52:28.948238 (+ 278us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:52:28.948344 (+ 106us) client_negotiation.cc:769] Sending connection context
0620 14:52:28.948439 (+ 95us) client_negotiation.cc:240] Negotiation successful
0620 14:52:28.948635 (+ 196us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1013,"mutex_wait_us":338,"thread_start_us":807,"threads_started":1}
I20260620 14:52:28.950256 20681 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "710f5da9fa394eb5affbac105d6f6ae1" candidate_uuid: "de124445be1e46b88cb67e1db82518aa" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8de497f6aea743a1862bf9c6a91b4a2c" is_pre_election: true
I20260620 14:52:28.950517 20420 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "710f5da9fa394eb5affbac105d6f6ae1" candidate_uuid: "de124445be1e46b88cb67e1db82518aa" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "18978e1b9066425cabc3a507e86915d5" is_pre_election: true
I20260620 14:52:28.950794 20681 raft_consensus.cc:2454] T 710f5da9fa394eb5affbac105d6f6ae1 P 8de497f6aea743a1862bf9c6a91b4a2c [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate de124445be1e46b88cb67e1db82518aa in term 0.
I20260620 14:52:28.951073 20420 raft_consensus.cc:2454] T 710f5da9fa394eb5affbac105d6f6ae1 P 18978e1b9066425cabc3a507e86915d5 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate de124445be1e46b88cb67e1db82518aa in term 0.
I20260620 14:52:28.951627 20484 leader_election.cc:302] T 710f5da9fa394eb5affbac105d6f6ae1 P de124445be1e46b88cb67e1db82518aa [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 8de497f6aea743a1862bf9c6a91b4a2c, de124445be1e46b88cb67e1db82518aa; no voters:
I20260620 14:52:28.952168 20751 raft_consensus.cc:2781] T 710f5da9fa394eb5affbac105d6f6ae1 P de124445be1e46b88cb67e1db82518aa [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260620 14:52:28.952318 20751 raft_consensus.cc:482] T 710f5da9fa394eb5affbac105d6f6ae1 P de124445be1e46b88cb67e1db82518aa [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260620 14:52:28.952440 20751 raft_consensus.cc:3037] T 710f5da9fa394eb5affbac105d6f6ae1 P de124445be1e46b88cb67e1db82518aa [term 0 FOLLOWER]: Advancing to term 1
I20260620 14:52:28.954860 20751 raft_consensus.cc:504] T 710f5da9fa394eb5affbac105d6f6ae1 P de124445be1e46b88cb67e1db82518aa [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8de497f6aea743a1862bf9c6a91b4a2c" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 44141 } } peers { permanent_uuid: "18978e1b9066425cabc3a507e86915d5" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 36767 } } peers { permanent_uuid: "de124445be1e46b88cb67e1db82518aa" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 41915 } }
I20260620 14:52:28.955518 20751 leader_election.cc:288] T 710f5da9fa394eb5affbac105d6f6ae1 P de124445be1e46b88cb67e1db82518aa [CANDIDATE]: Term 1 election: Requested vote from peers 8de497f6aea743a1862bf9c6a91b4a2c (127.0.97.3:44141), 18978e1b9066425cabc3a507e86915d5 (127.0.97.1:36767)
I20260620 14:52:28.956017 20681 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "710f5da9fa394eb5affbac105d6f6ae1" candidate_uuid: "de124445be1e46b88cb67e1db82518aa" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8de497f6aea743a1862bf9c6a91b4a2c"
I20260620 14:52:28.956099 20420 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "710f5da9fa394eb5affbac105d6f6ae1" candidate_uuid: "de124445be1e46b88cb67e1db82518aa" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "18978e1b9066425cabc3a507e86915d5"
I20260620 14:52:28.956262 20681 raft_consensus.cc:3037] T 710f5da9fa394eb5affbac105d6f6ae1 P 8de497f6aea743a1862bf9c6a91b4a2c [term 0 FOLLOWER]: Advancing to term 1
I20260620 14:52:28.956362 20420 raft_consensus.cc:3037] T 710f5da9fa394eb5affbac105d6f6ae1 P 18978e1b9066425cabc3a507e86915d5 [term 0 FOLLOWER]: Advancing to term 1
I20260620 14:52:28.958642 20420 raft_consensus.cc:2454] T 710f5da9fa394eb5affbac105d6f6ae1 P 18978e1b9066425cabc3a507e86915d5 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate de124445be1e46b88cb67e1db82518aa in term 1.
I20260620 14:52:28.958647 20681 raft_consensus.cc:2454] T 710f5da9fa394eb5affbac105d6f6ae1 P 8de497f6aea743a1862bf9c6a91b4a2c [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate de124445be1e46b88cb67e1db82518aa in term 1.
I20260620 14:52:28.959228 20486 leader_election.cc:302] T 710f5da9fa394eb5affbac105d6f6ae1 P de124445be1e46b88cb67e1db82518aa [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 18978e1b9066425cabc3a507e86915d5, de124445be1e46b88cb67e1db82518aa; no voters:
I20260620 14:52:28.959555 20751 raft_consensus.cc:2781] T 710f5da9fa394eb5affbac105d6f6ae1 P de124445be1e46b88cb67e1db82518aa [term 1 FOLLOWER]: Leader election won for term 1
I20260620 14:52:28.960039 20751 raft_consensus.cc:686] T 710f5da9fa394eb5affbac105d6f6ae1 P de124445be1e46b88cb67e1db82518aa [term 1 LEADER]: Becoming Leader. State: Replica: de124445be1e46b88cb67e1db82518aa, State: Running, Role: LEADER
I20260620 14:52:28.960582 20751 consensus_queue.cc:237] T 710f5da9fa394eb5affbac105d6f6ae1 P de124445be1e46b88cb67e1db82518aa [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8de497f6aea743a1862bf9c6a91b4a2c" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 44141 } } peers { permanent_uuid: "18978e1b9066425cabc3a507e86915d5" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 36767 } } peers { permanent_uuid: "de124445be1e46b88cb67e1db82518aa" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 41915 } }
I20260620 14:52:28.965739 20277 catalog_manager.cc:5495] T 710f5da9fa394eb5affbac105d6f6ae1 P de124445be1e46b88cb67e1db82518aa reported cstate change: term changed from 0 to 1, leader changed from <none> to de124445be1e46b88cb67e1db82518aa (127.0.97.2). New cstate: current_term: 1 leader_uuid: "de124445be1e46b88cb67e1db82518aa" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8de497f6aea743a1862bf9c6a91b4a2c" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 44141 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "18978e1b9066425cabc3a507e86915d5" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 36767 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "de124445be1e46b88cb67e1db82518aa" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 41915 } health_report { overall_health: HEALTHY } } }
I20260620 14:52:28.996027 20743 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:52:28.990126 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42886 (local address 127.0.97.2:41915)
0620 14:52:28.990329 (+ 203us) server_negotiation.cc:206] Beginning negotiation
0620 14:52:28.990335 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 14:52:28.990452 (+ 117us) server_negotiation.cc:407] Connection header received
0620 14:52:28.990720 (+ 268us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:52:28.990725 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:52:28.990828 (+ 103us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:52:28.990982 (+ 154us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 14:52:28.991341 (+ 359us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:52:28.992078 (+ 737us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:52:28.993759 (+ 1681us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:52:28.994252 (+ 493us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:52:28.994378 (+ 126us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:52:28.994824 (+ 446us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 14:52:28.995353 (+ 529us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0620 14:52:28.995475 (+ 122us) server_negotiation.cc:1035] Waiting for connection context
0620 14:52:28.995578 (+ 103us) server_negotiation.cc:299] Negotiation successful
0620 14:52:28.995704 (+ 126us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":56}
W20260620 14:52:29.002102 20466 tablet.cc:2369] T 710f5da9fa394eb5affbac105d6f6ae1 P 18978e1b9066425cabc3a507e86915d5: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260620 14:52:29.002619 20597 tablet.cc:2369] T 710f5da9fa394eb5affbac105d6f6ae1 P de124445be1e46b88cb67e1db82518aa: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260620 14:52:29.004813 20420 raft_consensus.cc:1261] T 710f5da9fa394eb5affbac105d6f6ae1 P 18978e1b9066425cabc3a507e86915d5 [term 1 FOLLOWER]: Refusing update from remote peer de124445be1e46b88cb67e1db82518aa: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 14:52:29.004814 20681 raft_consensus.cc:1261] T 710f5da9fa394eb5affbac105d6f6ae1 P 8de497f6aea743a1862bf9c6a91b4a2c [term 1 FOLLOWER]: Refusing update from remote peer de124445be1e46b88cb67e1db82518aa: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 14:52:29.005928 20751 consensus_queue.cc:1035] T 710f5da9fa394eb5affbac105d6f6ae1 P de124445be1e46b88cb67e1db82518aa [LEADER]: Connected to new peer: Peer: permanent_uuid: "18978e1b9066425cabc3a507e86915d5" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 36767 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 14:52:29.006229 20756 consensus_queue.cc:1035] T 710f5da9fa394eb5affbac105d6f6ae1 P de124445be1e46b88cb67e1db82518aa [LEADER]: Connected to new peer: Peer: permanent_uuid: "8de497f6aea743a1862bf9c6a91b4a2c" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 44141 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20260620 14:52:29.019107 20727 tablet.cc:2369] T 710f5da9fa394eb5affbac105d6f6ae1 P 8de497f6aea743a1862bf9c6a91b4a2c: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260620 14:52:29.020437 20759 mvcc.cc:205] Tried to move back new op lower bound from 7298937442312400896 to 7298937442146942976. Current Snapshot: MvccSnapshot[applied={T|T < 7298937442312400896}]
I20260620 14:52:29.039052 20277 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:60334:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 14:52:29.039388 20277 catalog_manager.cc:2645] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:60334:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 14:52:29.043115 20277 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 937605b6de7d4df4a16f746ca2dd382c: Sending DeleteTablet for 3 replicas of tablet 710f5da9fa394eb5affbac105d6f6ae1
I20260620 14:52:29.044433 20661 tablet_service.cc:1514] Processing DeleteTablet for tablet 710f5da9fa394eb5affbac105d6f6ae1 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 14:52:29 UTC) from {username='slave'} at 127.0.0.1:36816
I20260620 14:52:29.044562 20400 tablet_service.cc:1514] Processing DeleteTablet for tablet 710f5da9fa394eb5affbac105d6f6ae1 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 14:52:29 UTC) from {username='slave'} at 127.0.0.1:45996
I20260620 14:52:29.044826 20531 tablet_service.cc:1514] Processing DeleteTablet for tablet 710f5da9fa394eb5affbac105d6f6ae1 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 14:52:29 UTC) from {username='slave'} at 127.0.0.1:42874
I20260620 14:52:29.049832 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskUci3DU/build/asan/bin/kudu pid 20338
I20260620 14:52:29.057404 20769 tablet_replica.cc:330] T 710f5da9fa394eb5affbac105d6f6ae1 P 8de497f6aea743a1862bf9c6a91b4a2c: stopping tablet replica
I20260620 14:52:29.058138 20769 raft_consensus.cc:2229] T 710f5da9fa394eb5affbac105d6f6ae1 P 8de497f6aea743a1862bf9c6a91b4a2c [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 14:52:29.058856 20769 raft_consensus.cc:2258] T 710f5da9fa394eb5affbac105d6f6ae1 P 8de497f6aea743a1862bf9c6a91b4a2c [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 14:52:29.061502 20769 ts_tablet_manager.cc:1905] T 710f5da9fa394eb5affbac105d6f6ae1 P 8de497f6aea743a1862bf9c6a91b4a2c: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 14:52:29.069043 20769 ts_tablet_manager.cc:1918] T 710f5da9fa394eb5affbac105d6f6ae1 P 8de497f6aea743a1862bf9c6a91b4a2c: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260620 14:52:29.069388 20769 log.cc:1196] T 710f5da9fa394eb5affbac105d6f6ae1 P 8de497f6aea743a1862bf9c6a91b4a2c: Deleting WAL directory at /tmp/dist-test-taskUci3DU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781966829091300-388-0/minicluster-data/ts-2/wal/wals/710f5da9fa394eb5affbac105d6f6ae1
I20260620 14:52:29.070181 20769 ts_tablet_manager.cc:1939] T 710f5da9fa394eb5affbac105d6f6ae1 P 8de497f6aea743a1862bf9c6a91b4a2c: Deleting consensus metadata
I20260620 14:52:29.072487 20263 catalog_manager.cc:4841] TS 8de497f6aea743a1862bf9c6a91b4a2c (127.0.97.3:44141): tablet 710f5da9fa394eb5affbac105d6f6ae1 (table test-table [id=da48a8822e614e3b88479de387dfb875]) successfully deleted
I20260620 14:52:29.073647 20770 tablet_replica.cc:330] T 710f5da9fa394eb5affbac105d6f6ae1 P 18978e1b9066425cabc3a507e86915d5: stopping tablet replica
I20260620 14:52:29.074242 20770 raft_consensus.cc:2229] T 710f5da9fa394eb5affbac105d6f6ae1 P 18978e1b9066425cabc3a507e86915d5 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 14:52:29.074980 20770 raft_consensus.cc:2258] T 710f5da9fa394eb5affbac105d6f6ae1 P 18978e1b9066425cabc3a507e86915d5 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 14:52:29.075896 20771 tablet_replica.cc:330] T 710f5da9fa394eb5affbac105d6f6ae1 P de124445be1e46b88cb67e1db82518aa: stopping tablet replica
I20260620 14:52:29.076423 20771 raft_consensus.cc:2229] T 710f5da9fa394eb5affbac105d6f6ae1 P de124445be1e46b88cb67e1db82518aa [term 1 LEADER]: Raft consensus shutting down.
I20260620 14:52:29.077363 20771 raft_consensus.cc:2258] T 710f5da9fa394eb5affbac105d6f6ae1 P de124445be1e46b88cb67e1db82518aa [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 14:52:29.077376 20770 ts_tablet_manager.cc:1905] T 710f5da9fa394eb5affbac105d6f6ae1 P 18978e1b9066425cabc3a507e86915d5: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 14:52:29.079990 20771 ts_tablet_manager.cc:1905] T 710f5da9fa394eb5affbac105d6f6ae1 P de124445be1e46b88cb67e1db82518aa: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 14:52:29.080322 20742 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:52:29.051552 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46006 (local address 127.0.97.1:36767)
0620 14:52:29.051815 (+ 263us) server_negotiation.cc:206] Beginning negotiation
0620 14:52:29.051825 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0620 14:52:29.069707 (+ 17882us) server_negotiation.cc:407] Connection header received
0620 14:52:29.070428 (+ 721us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:52:29.070438 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:52:29.070624 (+ 186us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:52:29.070726 (+ 102us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:52:29.071711 (+ 985us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:52:29.073265 (+ 1554us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:52:29.075902 (+ 2637us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:52:29.076570 (+ 668us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:52:29.076680 (+ 110us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:52:29.078319 (+ 1639us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 14:52:29.078333 (+ 14us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 14:52:29.078356 (+ 23us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 14:52:29.079184 (+ 828us) server_negotiation.cc:922] calling sasl_server_start()
0620 14:52:29.079254 (+ 70us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 14:52:29.079283 (+ 29us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 14:52:29.079419 (+ 136us) server_negotiation.cc:1035] Waiting for connection context
0620 14:52:29.079762 (+ 343us) server_negotiation.cc:299] Negotiation successful
0620 14:52:29.079901 (+ 139us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":97}
I20260620 14:52:29.081625 20460 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 14:52:29.088308 20771 ts_tablet_manager.cc:1918] T 710f5da9fa394eb5affbac105d6f6ae1 P de124445be1e46b88cb67e1db82518aa: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260620 14:52:29.088611 20771 log.cc:1196] T 710f5da9fa394eb5affbac105d6f6ae1 P de124445be1e46b88cb67e1db82518aa: Deleting WAL directory at /tmp/dist-test-taskUci3DU/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781966829091300-388-0/minicluster-data/ts-1/wal/wals/710f5da9fa394eb5affbac105d6f6ae1
I20260620 14:52:29.089543 20771 ts_tablet_manager.cc:1939] T 710f5da9fa394eb5affbac105d6f6ae1 P de124445be1e46b88cb67e1db82518aa: Deleting consensus metadata
I20260620 14:52:29.092435 20265 catalog_manager.cc:4841] TS de124445be1e46b88cb67e1db82518aa (127.0.97.2:41915): tablet 710f5da9fa394eb5affbac105d6f6ae1 (table test-table [id=da48a8822e614e3b88479de387dfb875]) successfully deleted
I20260620 14:52:29.240105 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskUci3DU/build/asan/bin/kudu with pid 20338
I20260620 14:52:29.259032 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskUci3DU/build/asan/bin/kudu pid 20469
W20260620 14:52:29.259475 20265 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.0.97.1:36767 (error 108)
I20260620 14:52:29.261737 20739 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:52:29.261222 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:36767 (local address 127.0.0.1:46010)
0620 14:52:29.261418 (+ 196us) negotiation.cc:107] Waiting for socket to connect
0620 14:52:29.261542 (+ 124us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.1:36767: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":69}
W20260620 14:52:29.262104 20265 catalog_manager.cc:4584] TS 18978e1b9066425cabc3a507e86915d5 (127.0.97.1:36767): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 710f5da9fa394eb5affbac105d6f6ae1: Network error: Client connection negotiation failed: client connection to 127.0.97.1:36767: connect: Connection refused (error 111)
I20260620 14:52:29.269412 20743 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:52:29.260894 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42900 (local address 127.0.97.2:41915)
0620 14:52:29.261099 (+ 205us) server_negotiation.cc:206] Beginning negotiation
0620 14:52:29.261106 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0620 14:52:29.261679 (+ 573us) server_negotiation.cc:407] Connection header received
0620 14:52:29.262630 (+ 951us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:52:29.262644 (+ 14us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:52:29.262733 (+ 89us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:52:29.262879 (+ 146us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:52:29.263302 (+ 423us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:52:29.264299 (+ 997us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:52:29.265913 (+ 1614us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:52:29.266391 (+ 478us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:52:29.266544 (+ 153us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:52:29.267767 (+ 1223us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 14:52:29.267778 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 14:52:29.267793 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 14:52:29.268453 (+ 660us) server_negotiation.cc:922] calling sasl_server_start()
0620 14:52:29.268509 (+ 56us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 14:52:29.268531 (+ 22us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 14:52:29.268684 (+ 153us) server_negotiation.cc:1035] Waiting for connection context
0620 14:52:29.268822 (+ 138us) server_negotiation.cc:299] Negotiation successful
0620 14:52:29.268942 (+ 120us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":51}
I20260620 14:52:29.270155 20591 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 14:52:29.316186 20739 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:52:29.315655 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:36767 (local address 127.0.0.1:46022)
0620 14:52:29.315890 (+ 235us) negotiation.cc:107] Waiting for socket to connect
0620 14:52:29.316011 (+ 121us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.1:36767: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":84}
I20260620 14:52:29.317409 20739 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:52:29.317001 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:36767 (local address 127.0.0.1:46028)
0620 14:52:29.317168 (+ 167us) negotiation.cc:107] Waiting for socket to connect
0620 14:52:29.317282 (+ 114us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.1:36767: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":61}
I20260620 14:52:29.377974 20739 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:52:29.377416 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:36767 (local address 127.0.0.1:46030)
0620 14:52:29.377654 (+ 238us) negotiation.cc:107] Waiting for socket to connect
0620 14:52:29.377807 (+ 153us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.1:36767: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":96}
I20260620 14:52:29.379176 20739 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:52:29.378806 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:36767 (local address 127.0.0.1:46044)
0620 14:52:29.378967 (+ 161us) negotiation.cc:107] Waiting for socket to connect
0620 14:52:29.379059 (+ 92us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.1:36767: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":51}
I20260620 14:52:29.387681 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskUci3DU/build/asan/bin/kudu with pid 20469
I20260620 14:52:29.407579 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskUci3DU/build/asan/bin/kudu pid 20599
I20260620 14:52:29.414326 20741 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:52:29.408475 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36822 (local address 127.0.97.3:44141)
0620 14:52:29.408657 (+ 182us) server_negotiation.cc:206] Beginning negotiation
0620 14:52:29.408661 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 14:52:29.408684 (+ 23us) server_negotiation.cc:407] Connection header received
0620 14:52:29.408862 (+ 178us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:52:29.408865 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:52:29.408951 (+ 86us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:52:29.409051 (+ 100us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:52:29.409408 (+ 357us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:52:29.410126 (+ 718us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:52:29.411372 (+ 1246us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:52:29.411874 (+ 502us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:52:29.411979 (+ 105us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:52:29.413064 (+ 1085us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 14:52:29.413075 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 14:52:29.413090 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 14:52:29.413669 (+ 579us) server_negotiation.cc:922] calling sasl_server_start()
0620 14:52:29.413721 (+ 52us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 14:52:29.413741 (+ 20us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 14:52:29.413851 (+ 110us) server_negotiation.cc:1035] Waiting for connection context
0620 14:52:29.413958 (+ 107us) server_negotiation.cc:299] Negotiation successful
0620 14:52:29.414057 (+ 99us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":65}
I20260620 14:52:29.415014 20721 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 14:52:29.459972 20739 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:52:29.459310 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:36767 (local address 127.0.0.1:39368)
0620 14:52:29.459627 (+ 317us) negotiation.cc:107] Waiting for socket to connect
0620 14:52:29.459749 (+ 122us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.1:36767: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":115}
I20260620 14:52:29.461234 20739 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:52:29.460716 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:36767 (local address 127.0.0.1:39370)
0620 14:52:29.460959 (+ 243us) negotiation.cc:107] Waiting for socket to connect
0620 14:52:29.461059 (+ 100us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.1:36767: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":124}
I20260620 14:52:29.505914 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskUci3DU/build/asan/bin/kudu with pid 20599
I20260620 14:52:29.522881 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskUci3DU/build/asan/bin/kudu pid 20248
I20260620 14:52:29.524129 20309 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 14:52:29.637440 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskUci3DU/build/asan/bin/kudu with pid 20248
2026-06-20T14:52:29Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/5 (3053 ms)
[----------] 6 tests from EncryptionPolicyTest (22427 ms total)
[----------] 18 tests from AuthTokenIssuingTest
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/0
Loading random data
Initializing database '/tmp/dist-test-taskUci3DU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1781966829091300-388-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 20 14:52:29 dist-test-slave-q9j6 krb5kdc[20780](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 20 14:52:29 dist-test-slave-q9j6 krb5kdc[20780](info): set up 2 sockets
Jun 20 14:52:29 dist-test-slave-q9j6 krb5kdc[20780](info): commencing operation
krb5kdc: starting...
W20260620 14:52:30.764320 388 mini_kdc.cc:121] Time spent starting KDC: real 1.081s user 0.005s sys 0.024s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskUci3DU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1781966829091300-388-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskUci3DU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1781966829091300-388-0/krb5kdc/test-user.keytab.
Jun 20 14:52:30 dist-test-slave-q9j6 krb5kdc[20780](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781967150, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-20T14:52:30Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T14:52:30Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskUci3DU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1781966829091300-388-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskUci3DU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1781966829091300-388-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260620 14:52:31.035218 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskUci3DU/build/asan/bin/kudu
/tmp/dist-test-taskUci3DU/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskUci3DU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1781966829091300-388-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskUci3DU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1781966829091300-388-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskUci3DU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1781966829091300-388-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskUci3DU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1781966829091300-388-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:38099
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskUci3DU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1781966829091300-388-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:44855
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:38099
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskUci3DU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1781966829091300-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskUci3DU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1781966829091300-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskUci3DU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1781966829091300-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 14:52:31.391855 20793 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 14:52:31.392261 20793 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 14:52:31.392428 20793 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 14:52:31.400137 20793 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 14:52:31.400336 20793 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 14:52:31.400452 20793 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 14:52:31.400563 20793 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 14:52:31.400638 20793 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 14:52:31.411103 20793 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:44855
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskUci3DU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1781966829091300-388-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskUci3DU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1781966829091300-388-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:38099
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskUci3DU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1781966829091300-388-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:38099
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskUci3DU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1781966829091300-388-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskUci3DU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1781966829091300-388-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 13:41:20 UTC on ccycloud.quasar-vdagoi.root.comops.site
build id 4457462
ASAN enabled
I20260620 14:52:31.412680 20793 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 14:52:31.414115 20793 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 14:52:31.427663 20801 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 14:52:31.427765 20798 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 14:52:31.427739 20799 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 14:52:31.428798 20793 server_base.cc:1034] running on GCE node
I20260620 14:52:31.429723 20793 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 14:52:31.431377 20793 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 14:52:31.432597 20793 hybrid_clock.cc:648] HybridClock initialized: now 1781967151432520 us; error 83 us; skew 500 ppm
I20260620 14:52:31.433015 20793 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 14:52:31 dist-test-slave-q9j6 krb5kdc[20780](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781967151, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 14:52:31.436704 20793 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260620 14:52:31.438398 20793 webserver.cc:460] Webserver started at http://127.0.0.1:34337/ using document root <none> and password file <none>
I20260620 14:52:31.439213 20793 fs_manager.cc:359] Metadata directory not provided
I20260620 14:52:31.439363 20793 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 14:52:31.439806 20793 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 14:52:31.443174 20793 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskUci3DU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1781966829091300-388-0/minicluster-data/master-0/data/instance:
uuid: "9761f2e5c03149ad8b684f9c1c9b4152"
format_stamp: "Formatted at 2026-06-20 14:52:31 on dist-test-slave-q9j6"
server_key: "8c8a1f5661191f43d238eb142e7a929c"
server_key_iv: "dde29b84b5455b8ea1086bd61c819f6e"
server_key_version: "encryptionkey@0"
I20260620 14:52:31.444097 20793 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskUci3DU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1781966829091300-388-0/minicluster-data/master-0/wal/instance:
uuid: "9761f2e5c03149ad8b684f9c1c9b4152"
format_stamp: "Formatted at 2026-06-20 14:52:31 on dist-test-slave-q9j6"
server_key: "8c8a1f5661191f43d238eb142e7a929c"
server_key_iv: "dde29b84b5455b8ea1086bd61c819f6e"
server_key_version: "encryptionkey@0"
I20260620 14:52:31.449970 20793 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260620 14:52:31.454030 20808 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 14:52:31.455790 20793 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260620 14:52:31.456022 20793 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskUci3DU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1781966829091300-388-0/minicluster-data/master-0/data,/tmp/dist-test-taskUci3DU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1781966829091300-388-0/minicluster-data/master-0/wal
uuid: "9761f2e5c03149ad8b684f9c1c9b4152"
format_stamp: "Formatted at 2026-06-20 14:52:31 on dist-test-slave-q9j6"
server_key: "8c8a1f5661191f43d238eb142e7a929c"
server_key_iv: "dde29b84b5455b8ea1086bd61c819f6e"
server_key_version: "encryptionkey@0"
I20260620 14:52:31.456247 20793 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskUci3DU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1781966829091300-388-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskUci3DU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1781966829091300-388-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskUci3DU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1781966829091300-388-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 14:52:31.476717 20793 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 14:52:31.481220 20793 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 14:52:31.481586 20793 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 14:52:31.502044 20793 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:38099
I20260620 14:52:31.502063 20859 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:38099 every 8 connection(s)
I20260620 14:52:31.503636 20793 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskUci3DU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1781966829091300-388-0/minicluster-data/master-0/data/info.pb
I20260620 14:52:31.506719 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskUci3DU/build/asan/bin/kudu as pid 20793
I20260620 14:52:31.506994 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskUci3DU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1781966829091300-388-0/minicluster-data/master-0/wal/instance
I20260620 14:52:31.507932 388 external_mini_cluster.cc:1428] Setting key a6a0357c4b333569f812c13e0450b8b6
I20260620 14:52:31.508522 20860 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 14:52:31.521835 20860 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 9761f2e5c03149ad8b684f9c1c9b4152: Bootstrap starting.
Jun 20 14:52:31 dist-test-slave-q9j6 krb5kdc[20780](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781967150, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.0.1@KRBTEST.COM
I20260620 14:52:31.526207 20860 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 9761f2e5c03149ad8b684f9c1c9b4152: Neither blocks nor log segments found. Creating new log.
I20260620 14:52:31.527582 20860 log.cc:824] T 00000000000000000000000000000000 P 9761f2e5c03149ad8b684f9c1c9b4152: Log is configured to *not* fsync() on all Append() calls
I20260620 14:52:31.531466 20860 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 9761f2e5c03149ad8b684f9c1c9b4152: No bootstrap required, opened a new log
I20260620 14:52:31.533905 20863 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:52:31.510138 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47441 (local address 127.0.0.1:38099)
0620 14:52:31.510962 (+ 824us) server_negotiation.cc:206] Beginning negotiation
0620 14:52:31.510975 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0620 14:52:31.511021 (+ 46us) server_negotiation.cc:407] Connection header received
0620 14:52:31.513562 (+ 2541us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:52:31.513588 (+ 26us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:52:31.514086 (+ 498us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:52:31.514564 (+ 478us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:52:31.516233 (+ 1669us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:52:31.517539 (+ 1306us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:52:31.518848 (+ 1309us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:52:31.519694 (+ 846us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:52:31.519922 (+ 228us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:52:31.525018 (+ 5096us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 14:52:31.525028 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 14:52:31.525048 (+ 20us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 14:52:31.525095 (+ 47us) server_negotiation.cc:922] calling sasl_server_start()
0620 14:52:31.528982 (+ 3887us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:52:31.530207 (+ 1225us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:52:31.530214 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:52:31.530221 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:52:31.530409 (+ 188us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:52:31.530981 (+ 572us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:52:31.530984 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:52:31.530986 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:52:31.531567 (+ 581us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 14:52:31.531959 (+ 392us) server_negotiation.cc:1035] Waiting for connection context
0620 14:52:31.532482 (+ 523us) server_negotiation.cc:299] Negotiation successful
0620 14:52:31.532824 (+ 342us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":602,"thread_start_us":365,"threads_started":1}
I20260620 14:52:31.538419 388 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260620 14:52:31.540194 20860 raft_consensus.cc:348] T 00000000000000000000000000000000 P 9761f2e5c03149ad8b684f9c1c9b4152 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9761f2e5c03149ad8b684f9c1c9b4152" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 38099 } }
I20260620 14:52:31.540627 20860 raft_consensus.cc:374] T 00000000000000000000000000000000 P 9761f2e5c03149ad8b684f9c1c9b4152 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 14:52:31.540728 20860 raft_consensus.cc:729] T 00000000000000000000000000000000 P 9761f2e5c03149ad8b684f9c1c9b4152 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9761f2e5c03149ad8b684f9c1c9b4152, State: Initialized, Role: FOLLOWER
I20260620 14:52:31.541433 20860 consensus_queue.cc:260] T 00000000000000000000000000000000 P 9761f2e5c03149ad8b684f9c1c9b4152 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9761f2e5c03149ad8b684f9c1c9b4152" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 38099 } }
I20260620 14:52:31.541749 20860 raft_consensus.cc:388] T 00000000000000000000000000000000 P 9761f2e5c03149ad8b684f9c1c9b4152 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 14:52:31.541966 20860 raft_consensus.cc:482] T 00000000000000000000000000000000 P 9761f2e5c03149ad8b684f9c1c9b4152 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 14:52:31.542163 20860 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 9761f2e5c03149ad8b684f9c1c9b4152 [term 0 FOLLOWER]: Advancing to term 1
I20260620 14:52:31.545234 20860 raft_consensus.cc:504] T 00000000000000000000000000000000 P 9761f2e5c03149ad8b684f9c1c9b4152 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9761f2e5c03149ad8b684f9c1c9b4152" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 38099 } }
I20260620 14:52:31.545835 20860 leader_election.cc:302] T 00000000000000000000000000000000 P 9761f2e5c03149ad8b684f9c1c9b4152 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 9761f2e5c03149ad8b684f9c1c9b4152; no voters:
I20260620 14:52:31.546581 20860 leader_election.cc:288] T 00000000000000000000000000000000 P 9761f2e5c03149ad8b684f9c1c9b4152 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260620 14:52:31.546703 20870 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 9761f2e5c03149ad8b684f9c1c9b4152 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 14:52:31.549021 20860 sys_catalog.cc:564] T 00000000000000000000000000000000 P 9761f2e5c03149ad8b684f9c1c9b4152 [sys.catalog]: configured and running, proceeding with master startup.
I20260620 14:52:31.551973 20870 raft_consensus.cc:686] T 00000000000000000000000000000000 P 9761f2e5c03149ad8b684f9c1c9b4152 [term 1 LEADER]: Becoming Leader. State: Replica: 9761f2e5c03149ad8b684f9c1c9b4152, State: Running, Role: LEADER
I20260620 14:52:31.552804 20870 consensus_queue.cc:237] T 00000000000000000000000000000000 P 9761f2e5c03149ad8b684f9c1c9b4152 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9761f2e5c03149ad8b684f9c1c9b4152" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 38099 } }
I20260620 14:52:31.556217 20871 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9761f2e5c03149ad8b684f9c1c9b4152 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "9761f2e5c03149ad8b684f9c1c9b4152" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9761f2e5c03149ad8b684f9c1c9b4152" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 38099 } } }
I20260620 14:52:31.556675 20871 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9761f2e5c03149ad8b684f9c1c9b4152 [sys.catalog]: This master's current role is: LEADER
I20260620 14:52:31.562207 20872 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9761f2e5c03149ad8b684f9c1c9b4152 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 9761f2e5c03149ad8b684f9c1c9b4152. Latest consensus state: current_term: 1 leader_uuid: "9761f2e5c03149ad8b684f9c1c9b4152" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9761f2e5c03149ad8b684f9c1c9b4152" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 38099 } } }
I20260620 14:52:31.563434 20872 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9761f2e5c03149ad8b684f9c1c9b4152 [sys.catalog]: This master's current role is: LEADER
I20260620 14:52:31.564559 20863 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:52:31.543504 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33113 (local address 127.0.0.1:38099)
0620 14:52:31.543756 (+ 252us) server_negotiation.cc:206] Beginning negotiation
0620 14:52:31.543764 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0620 14:52:31.544196 (+ 432us) server_negotiation.cc:407] Connection header received
0620 14:52:31.544319 (+ 123us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:52:31.544328 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:52:31.544414 (+ 86us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:52:31.544532 (+ 118us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:52:31.546026 (+ 1494us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:52:31.547394 (+ 1368us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:52:31.549474 (+ 2080us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:52:31.550201 (+ 727us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:52:31.550369 (+ 168us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:52:31.554135 (+ 3766us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 14:52:31.554148 (+ 13us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 14:52:31.554156 (+ 8us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 14:52:31.554219 (+ 63us) server_negotiation.cc:922] calling sasl_server_start()
0620 14:52:31.559579 (+ 5360us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:52:31.561327 (+ 1748us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:52:31.561336 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:52:31.561342 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:52:31.561619 (+ 277us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:52:31.562415 (+ 796us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:52:31.562420 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:52:31.562424 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:52:31.562914 (+ 490us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 14:52:31.563876 (+ 962us) server_negotiation.cc:1035] Waiting for connection context
0620 14:52:31.564110 (+ 234us) server_negotiation.cc:299] Negotiation successful
0620 14:52:31.564314 (+ 204us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":71}
I20260620 14:52:31.569460 20876 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 14:52:31.575511 20876 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 14:52:31.586805 20876 catalog_manager.cc:1347] Generated new cluster ID: a46e38738d944b09ad171094cdcceccf
I20260620 14:52:31.586951 20876 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 14:52:31.627192 20876 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 14:52:31.628660 20876 catalog_manager.cc:1504] Loading token signing keys...
I20260620 14:52:31.644972 20876 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 9761f2e5c03149ad8b684f9c1c9b4152: Generated new TSK 0
I20260620 14:52:31.645939 20876 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260620 14:52:31.657961 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskUci3DU/build/asan/bin/kudu pid 20793
I20260620 14:52:31.659667 20855 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 14:52:31.755708 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskUci3DU/build/asan/bin/kudu with pid 20793
W20260620 14:52:31.773515 20778 connection.cc:535] client connection to 127.0.0.1:38099 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:38099): Connection reset by peer (error 104)
2026-06-20T14:52:31Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/0 (2110 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/1
Loading random data
Initializing database '/tmp/dist-test-taskUci3DU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1781966829091300-388-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 20 14:52:31 dist-test-slave-q9j6 krb5kdc[20892](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 20 14:52:31 dist-test-slave-q9j6 krb5kdc[20892](info): set up 2 sockets
Jun 20 14:52:31 dist-test-slave-q9j6 krb5kdc[20892](info): commencing operation
krb5kdc: starting...
W20260620 14:52:32.864037 388 mini_kdc.cc:121] Time spent starting KDC: real 1.071s user 0.003s sys 0.027s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskUci3DU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1781966829091300-388-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskUci3DU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1781966829091300-388-0/krb5kdc/test-user.keytab.
Jun 20 14:52:33 dist-test-slave-q9j6 krb5kdc[20892](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781967153, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-20T14:52:33Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T14:52:33Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskUci3DU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1781966829091300-388-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskUci3DU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1781966829091300-388-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260620 14:52:33.131373 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskUci3DU/build/asan/bin/kudu
/tmp/dist-test-taskUci3DU/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskUci3DU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1781966829091300-388-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskUci3DU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1781966829091300-388-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskUci3DU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1781966829091300-388-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskUci3DU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1781966829091300-388-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:41371
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskUci3DU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1781966829091300-388-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:41381
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:41371
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskUci3DU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1781966829091300-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskUci3DU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1781966829091300-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskUci3DU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1781966829091300-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 14:52:33.485105 20905 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 14:52:33.485507 20905 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 14:52:33.485682 20905 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 14:52:33.493317 20905 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 14:52:33.493521 20905 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 14:52:33.493642 20905 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 14:52:33.493757 20905 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 14:52:33.493834 20905 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 14:52:33.504463 20905 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:41381
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskUci3DU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1781966829091300-388-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskUci3DU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1781966829091300-388-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:41371
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskUci3DU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1781966829091300-388-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:41371
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskUci3DU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1781966829091300-388-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskUci3DU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1781966829091300-388-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 13:41:20 UTC on ccycloud.quasar-vdagoi.root.comops.site
build id 4457462
ASAN enabled
I20260620 14:52:33.505935 20905 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 14:52:33.507398 20905 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 14:52:33.520076 20910 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 14:52:33.520079 20913 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 14:52:33.520102 20911 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 14:52:33.521085 20905 server_base.cc:1034] running on GCE node
I20260620 14:52:33.521946 20905 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 14:52:33.523496 20905 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 14:52:33.524747 20905 hybrid_clock.cc:648] HybridClock initialized: now 1781967153524684 us; error 63 us; skew 500 ppm
I20260620 14:52:33.525135 20905 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 14:52:33 dist-test-slave-q9j6 krb5kdc[20892](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781967153, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 14:52:33.533092 20905 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260620 14:52:33.534905 20905 webserver.cc:460] Webserver started at http://127.0.0.1:34397/ using document root <none> and password file <none>
I20260620 14:52:33.535820 20905 fs_manager.cc:359] Metadata directory not provided
I20260620 14:52:33.535990 20905 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 14:52:33.536341 20905 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 14:52:33.539698 20905 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskUci3DU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1781966829091300-388-0/minicluster-data/master-0/data/instance:
uuid: "4cec98da86974fc1a43774ee8ccc86b6"
format_stamp: "Formatted at 2026-06-20 14:52:33 on dist-test-slave-q9j6"
server_key: "9c4b591c6efc1883e399f89e2efa0446"
server_key_iv: "e1bded0016327e6886f74f2cb53248d8"
server_key_version: "encryptionkey@0"
I20260620 14:52:33.540676 20905 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskUci3DU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1781966829091300-388-0/minicluster-data/master-0/wal/instance:
uuid: "4cec98da86974fc1a43774ee8ccc86b6"
format_stamp: "Formatted at 2026-06-20 14:52:33 on dist-test-slave-q9j6"
server_key: "9c4b591c6efc1883e399f89e2efa0446"
server_key_iv: "e1bded0016327e6886f74f2cb53248d8"
server_key_version: "encryptionkey@0"
I20260620 14:52:33.546442 20905 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260620 14:52:33.550491 20920 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 14:52:33.552201 20905 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260620 14:52:33.552428 20905 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskUci3DU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1781966829091300-388-0/minicluster-data/master-0/data,/tmp/dist-test-taskUci3DU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1781966829091300-388-0/minicluster-data/master-0/wal
uuid: "4cec98da86974fc1a43774ee8ccc86b6"
format_stamp: "Formatted at 2026-06-20 14:52:33 on dist-test-slave-q9j6"
server_key: "9c4b591c6efc1883e399f89e2efa0446"
server_key_iv: "e1bded0016327e6886f74f2cb53248d8"
server_key_version: "encryptionkey@0"
I20260620 14:52:33.552667 20905 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskUci3DU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1781966829091300-388-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskUci3DU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1781966829091300-388-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskUci3DU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1781966829091300-388-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 14:52:33.575104 20905 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 14:52:33.579770 20905 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 14:52:33.580124 20905 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 14:52:33.600145 20905 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:41371
I20260620 14:52:33.600186 20971 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:41371 every 8 connection(s)
I20260620 14:52:33.601626 20905 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskUci3DU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1781966829091300-388-0/minicluster-data/master-0/data/info.pb
I20260620 14:52:33.605772 20972 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 14:52:33.608358 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskUci3DU/build/asan/bin/kudu as pid 20905
I20260620 14:52:33.608567 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskUci3DU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1781966829091300-388-0/minicluster-data/master-0/wal/instance
I20260620 14:52:33.608995 388 external_mini_cluster.cc:1428] Setting key b661733644d632a9c9b3d2b404d02e6c
I20260620 14:52:33.618932 20972 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 4cec98da86974fc1a43774ee8ccc86b6: Bootstrap starting.
I20260620 14:52:33.622840 20972 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 4cec98da86974fc1a43774ee8ccc86b6: Neither blocks nor log segments found. Creating new log.
I20260620 14:52:33.623814 20975 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:52:33.611022 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41131 (local address 127.0.0.1:41371)
0620 14:52:33.611987 (+ 965us) server_negotiation.cc:206] Beginning negotiation
0620 14:52:33.612005 (+ 18us) server_negotiation.cc:399] Waiting for connection header
0620 14:52:33.612056 (+ 51us) server_negotiation.cc:407] Connection header received
0620 14:52:33.614002 (+ 1946us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:52:33.614029 (+ 27us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:52:33.614527 (+ 498us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:52:33.614983 (+ 456us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:52:33.615675 (+ 692us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:52:33.617106 (+ 1431us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:52:33.618573 (+ 1467us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:52:33.619410 (+ 837us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:52:33.619654 (+ 244us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:52:33.621145 (+ 1491us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 14:52:33.621154 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 14:52:33.621171 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 14:52:33.621775 (+ 604us) server_negotiation.cc:922] calling sasl_server_start()
0620 14:52:33.621866 (+ 91us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 14:52:33.621896 (+ 30us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 14:52:33.622081 (+ 185us) server_negotiation.cc:1035] Waiting for connection context
0620 14:52:33.622403 (+ 322us) server_negotiation.cc:299] Negotiation successful
0620 14:52:33.622582 (+ 179us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":711,"thread_start_us":410,"threads_started":1}
I20260620 14:52:33.624152 20972 log.cc:824] T 00000000000000000000000000000000 P 4cec98da86974fc1a43774ee8ccc86b6: Log is configured to *not* fsync() on all Append() calls
I20260620 14:52:33.628160 20972 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 4cec98da86974fc1a43774ee8ccc86b6: No bootstrap required, opened a new log
I20260620 14:52:33.628607 388 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260620 14:52:33.637274 20972 raft_consensus.cc:348] T 00000000000000000000000000000000 P 4cec98da86974fc1a43774ee8ccc86b6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4cec98da86974fc1a43774ee8ccc86b6" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 41371 } }
I20260620 14:52:33.637779 20972 raft_consensus.cc:374] T 00000000000000000000000000000000 P 4cec98da86974fc1a43774ee8ccc86b6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 14:52:33.637905 20972 raft_consensus.cc:729] T 00000000000000000000000000000000 P 4cec98da86974fc1a43774ee8ccc86b6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4cec98da86974fc1a43774ee8ccc86b6, State: Initialized, Role: FOLLOWER
I20260620 14:52:33.638715 20972 consensus_queue.cc:260] T 00000000000000000000000000000000 P 4cec98da86974fc1a43774ee8ccc86b6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4cec98da86974fc1a43774ee8ccc86b6" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 41371 } }
I20260620 14:52:33.638988 20972 raft_consensus.cc:388] T 00000000000000000000000000000000 P 4cec98da86974fc1a43774ee8ccc86b6 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 14:52:33.639153 20972 raft_consensus.cc:482] T 00000000000000000000000000000000 P 4cec98da86974fc1a43774ee8ccc86b6 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 14:52:33.639353 20972 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 4cec98da86974fc1a43774ee8ccc86b6 [term 0 FOLLOWER]: Advancing to term 1
I20260620 14:52:33.640712 20975 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:52:33.633187 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56931 (local address 127.0.0.1:41371)
0620 14:52:33.633407 (+ 220us) server_negotiation.cc:206] Beginning negotiation
0620 14:52:33.633414 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0620 14:52:33.633812 (+ 398us) server_negotiation.cc:407] Connection header received
0620 14:52:33.634001 (+ 189us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:52:33.634007 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:52:33.634110 (+ 103us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:52:33.634261 (+ 151us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:52:33.634938 (+ 677us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:52:33.636198 (+ 1260us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:52:33.637643 (+ 1445us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:52:33.638350 (+ 707us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:52:33.638476 (+ 126us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:52:33.639743 (+ 1267us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 14:52:33.639749 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 14:52:33.639754 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 14:52:33.639779 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0620 14:52:33.639850 (+ 71us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 14:52:33.639875 (+ 25us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 14:52:33.640000 (+ 125us) server_negotiation.cc:1035] Waiting for connection context
0620 14:52:33.640183 (+ 183us) server_negotiation.cc:299] Negotiation successful
0620 14:52:33.640253 (+ 70us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":68}
I20260620 14:52:33.642345 20972 raft_consensus.cc:504] T 00000000000000000000000000000000 P 4cec98da86974fc1a43774ee8ccc86b6 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4cec98da86974fc1a43774ee8ccc86b6" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 41371 } }
I20260620 14:52:33.642927 20972 leader_election.cc:302] T 00000000000000000000000000000000 P 4cec98da86974fc1a43774ee8ccc86b6 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 4cec98da86974fc1a43774ee8ccc86b6; no voters:
I20260620 14:52:33.643541 20972 leader_election.cc:288] T 00000000000000000000000000000000 P 4cec98da86974fc1a43774ee8ccc86b6 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260620 14:52:33.643751 20982 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 4cec98da86974fc1a43774ee8ccc86b6 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 14:52:33.644517 20982 raft_consensus.cc:686] T 00000000000000000000000000000000 P 4cec98da86974fc1a43774ee8ccc86b6 [term 1 LEADER]: Becoming Leader. State: Replica: 4cec98da86974fc1a43774ee8ccc86b6, State: Running, Role: LEADER
I20260620 14:52:33.645267 20982 consensus_queue.cc:237] T 00000000000000000000000000000000 P 4cec98da86974fc1a43774ee8ccc86b6 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4cec98da86974fc1a43774ee8ccc86b6" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 41371 } }
I20260620 14:52:33.646056 20972 sys_catalog.cc:564] T 00000000000000000000000000000000 P 4cec98da86974fc1a43774ee8ccc86b6 [sys.catalog]: configured and running, proceeding with master startup.
I20260620 14:52:33.648301 20983 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4cec98da86974fc1a43774ee8ccc86b6 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "4cec98da86974fc1a43774ee8ccc86b6" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4cec98da86974fc1a43774ee8ccc86b6" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 41371 } } }
I20260620 14:52:33.648702 20983 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4cec98da86974fc1a43774ee8ccc86b6 [sys.catalog]: This master's current role is: LEADER
I20260620 14:52:33.648433 20984 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4cec98da86974fc1a43774ee8ccc86b6 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 4cec98da86974fc1a43774ee8ccc86b6. Latest consensus state: current_term: 1 leader_uuid: "4cec98da86974fc1a43774ee8ccc86b6" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4cec98da86974fc1a43774ee8ccc86b6" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 41371 } } }
I20260620 14:52:33.648885 20984 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4cec98da86974fc1a43774ee8ccc86b6 [sys.catalog]: This master's current role is: LEADER
I20260620 14:52:33.649682 20989 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 14:52:33.661037 20989 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 14:52:33.672917 20989 catalog_manager.cc:1347] Generated new cluster ID: 06e2d5130a1f443b911a104c295e6fa9
I20260620 14:52:33.673187 20989 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 14:52:33.719677 20989 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 14:52:33.721112 20989 catalog_manager.cc:1504] Loading token signing keys...
I20260620 14:52:33.732594 20989 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 4cec98da86974fc1a43774ee8ccc86b6: Generated new TSK 0
I20260620 14:52:33.733531 20989 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260620 14:52:33.747146 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskUci3DU/build/asan/bin/kudu pid 20905
W20260620 14:52:33.748570 20967 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='slave'} at 127.0.0.1:41131
W20260620 14:52:33.749423 388 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskUci3DU/build/asan/bin/kudu pid 20905: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260620 14:52:33.749598 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskUci3DU/build/asan/bin/kudu with pid 20905
W20260620 14:52:33.767076 20890 connection.cc:535] client connection to 127.0.0.1:41371 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:41371): Connection reset by peer (error 104)
2026-06-20T14:52:33Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/1 (1993 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/2
Loading random data
Initializing database '/tmp/dist-test-taskUci3DU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1781966829091300-388-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 20 14:52:33 dist-test-slave-q9j6 krb5kdc[21003](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 20 14:52:33 dist-test-slave-q9j6 krb5kdc[21003](info): set up 2 sockets
Jun 20 14:52:33 dist-test-slave-q9j6 krb5kdc[21003](info): commencing operation
krb5kdc: starting...
W20260620 14:52:34.856138 388 mini_kdc.cc:121] Time spent starting KDC: real 1.069s user 0.002s sys 0.026s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskUci3DU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1781966829091300-388-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskUci3DU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1781966829091300-388-0/krb5kdc/test-user.keytab.
Jun 20 14:52:35 dist-test-slave-q9j6 krb5kdc[21003](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781967155, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-20T14:52:35Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T14:52:35Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskUci3DU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1781966829091300-388-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskUci3DU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1781966829091300-388-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260620 14:52:35.122227 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskUci3DU/build/asan/bin/kudu
/tmp/dist-test-taskUci3DU/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskUci3DU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1781966829091300-388-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskUci3DU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1781966829091300-388-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskUci3DU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1781966829091300-388-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskUci3DU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1781966829091300-388-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:46687
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskUci3DU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1781966829091300-388-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:34887
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:46687
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskUci3DU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1781966829091300-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskUci3DU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1781966829091300-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskUci3DU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1781966829091300-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 14:52:35.478196 21016 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 14:52:35.478601 21016 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 14:52:35.478785 21016 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 14:52:35.486979 21016 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 14:52:35.487167 21016 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 14:52:35.487289 21016 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 14:52:35.487404 21016 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 14:52:35.487495 21016 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 14:52:35.498265 21016 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:34887
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskUci3DU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1781966829091300-388-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskUci3DU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1781966829091300-388-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:46687
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskUci3DU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1781966829091300-388-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:46687
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskUci3DU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1781966829091300-388-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskUci3DU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1781966829091300-388-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 13:41:20 UTC on ccycloud.quasar-vdagoi.root.comops.site
build id 4457462
ASAN enabled
I20260620 14:52:35.499815 21016 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 14:52:35.501281 21016 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 14:52:35.514231 21021 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 14:52:35.514269 21024 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 14:52:35.514197 21022 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 14:52:35.515260 21016 server_base.cc:1034] running on GCE node
I20260620 14:52:35.516269 21016 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 14:52:35.517825 21016 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 14:52:35.519084 21016 hybrid_clock.cc:648] HybridClock initialized: now 1781967155519015 us; error 61 us; skew 500 ppm
I20260620 14:52:35.519474 21016 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 14:52:35 dist-test-slave-q9j6 krb5kdc[21003](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781967155, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 14:52:35.522992 21016 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260620 14:52:35.524752 21016 webserver.cc:460] Webserver started at http://127.0.0.1:34977/ using document root <none> and password file <none>
I20260620 14:52:35.525537 21016 fs_manager.cc:359] Metadata directory not provided
I20260620 14:52:35.525695 21016 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 14:52:35.526031 21016 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 14:52:35.529440 21016 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskUci3DU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1781966829091300-388-0/minicluster-data/master-0/data/instance:
uuid: "0b13ed10a2614be6bd0f4d539303a89f"
format_stamp: "Formatted at 2026-06-20 14:52:35 on dist-test-slave-q9j6"
server_key: "8667026685cebbe03841fa85648c4e38"
server_key_iv: "99a52fa8baf3baba9e629d637a137954"
server_key_version: "encryptionkey@0"
I20260620 14:52:35.530339 21016 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskUci3DU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1781966829091300-388-0/minicluster-data/master-0/wal/instance:
uuid: "0b13ed10a2614be6bd0f4d539303a89f"
format_stamp: "Formatted at 2026-06-20 14:52:35 on dist-test-slave-q9j6"
server_key: "8667026685cebbe03841fa85648c4e38"
server_key_iv: "99a52fa8baf3baba9e629d637a137954"
server_key_version: "encryptionkey@0"
I20260620 14:52:35.536417 21016 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.007s sys 0.000s
I20260620 14:52:35.540511 21031 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 14:52:35.542192 21016 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260620 14:52:35.542431 21016 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskUci3DU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1781966829091300-388-0/minicluster-data/master-0/data,/tmp/dist-test-taskUci3DU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1781966829091300-388-0/minicluster-data/master-0/wal
uuid: "0b13ed10a2614be6bd0f4d539303a89f"
format_stamp: "Formatted at 2026-06-20 14:52:35 on dist-test-slave-q9j6"
server_key: "8667026685cebbe03841fa85648c4e38"
server_key_iv: "99a52fa8baf3baba9e629d637a137954"
server_key_version: "encryptionkey@0"
I20260620 14:52:35.542685 21016 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskUci3DU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1781966829091300-388-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskUci3DU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1781966829091300-388-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskUci3DU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1781966829091300-388-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 14:52:35.549232 21016 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 14:52:35.554018 21016 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 14:52:35.554384 21016 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 14:52:35.574465 21016 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:46687
I20260620 14:52:35.574513 21082 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:46687 every 8 connection(s)
I20260620 14:52:35.576161 21016 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskUci3DU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1781966829091300-388-0/minicluster-data/master-0/data/info.pb
I20260620 14:52:35.580413 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskUci3DU/build/asan/bin/kudu as pid 21016
I20260620 14:52:35.580636 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskUci3DU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1781966829091300-388-0/minicluster-data/master-0/wal/instance
I20260620 14:52:35.580789 21083 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 14:52:35.581209 388 external_mini_cluster.cc:1428] Setting key ac4d284cafe491ca126bd0af4ea66412
I20260620 14:52:35.591224 21086 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:52:35.583289 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42153 (local address 127.0.0.1:46687)
0620 14:52:35.584277 (+ 988us) server_negotiation.cc:206] Beginning negotiation
0620 14:52:35.584301 (+ 24us) server_negotiation.cc:399] Waiting for connection header
0620 14:52:35.584367 (+ 66us) server_negotiation.cc:407] Connection header received
0620 14:52:35.586532 (+ 2165us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:52:35.586570 (+ 38us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:52:35.587162 (+ 592us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:52:35.587756 (+ 594us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:52:35.588921 (+ 1165us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 14:52:35.588929 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 14:52:35.588950 (+ 21us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 14:52:35.588971 (+ 21us) server_negotiation.cc:922] calling sasl_server_start()
0620 14:52:35.589056 (+ 85us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 14:52:35.589085 (+ 29us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 14:52:35.589362 (+ 277us) server_negotiation.cc:1035] Waiting for connection context
0620 14:52:35.589519 (+ 157us) server_negotiation.cc:299] Negotiation successful
0620 14:52:35.589784 (+ 265us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":685,"thread_start_us":433,"threads_started":1}
I20260620 14:52:35.595847 388 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260620 14:52:35.597527 21083 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 0b13ed10a2614be6bd0f4d539303a89f: Bootstrap starting.
I20260620 14:52:35.601456 21083 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 0b13ed10a2614be6bd0f4d539303a89f: Neither blocks nor log segments found. Creating new log.
I20260620 14:52:35.602782 21083 log.cc:824] T 00000000000000000000000000000000 P 0b13ed10a2614be6bd0f4d539303a89f: Log is configured to *not* fsync() on all Append() calls
I20260620 14:52:35.603184 21086 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:52:35.600599 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37863 (local address 127.0.0.1:46687)
0620 14:52:35.600788 (+ 189us) server_negotiation.cc:206] Beginning negotiation
0620 14:52:35.600795 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0620 14:52:35.601138 (+ 343us) server_negotiation.cc:407] Connection header received
0620 14:52:35.601313 (+ 175us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:52:35.601324 (+ 11us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:52:35.601398 (+ 74us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:52:35.601527 (+ 129us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:52:35.602548 (+ 1021us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 14:52:35.602554 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 14:52:35.602560 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 14:52:35.602582 (+ 22us) server_negotiation.cc:922] calling sasl_server_start()
0620 14:52:35.602640 (+ 58us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 14:52:35.602660 (+ 20us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 14:52:35.602796 (+ 136us) server_negotiation.cc:1035] Waiting for connection context
0620 14:52:35.602853 (+ 57us) server_negotiation.cc:299] Negotiation successful
0620 14:52:35.602917 (+ 64us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":70}
I20260620 14:52:35.606526 21083 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 0b13ed10a2614be6bd0f4d539303a89f: No bootstrap required, opened a new log
I20260620 14:52:35.615039 21083 raft_consensus.cc:348] T 00000000000000000000000000000000 P 0b13ed10a2614be6bd0f4d539303a89f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0b13ed10a2614be6bd0f4d539303a89f" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 46687 } }
I20260620 14:52:35.615511 21083 raft_consensus.cc:374] T 00000000000000000000000000000000 P 0b13ed10a2614be6bd0f4d539303a89f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 14:52:35.615676 21083 raft_consensus.cc:729] T 00000000000000000000000000000000 P 0b13ed10a2614be6bd0f4d539303a89f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0b13ed10a2614be6bd0f4d539303a89f, State: Initialized, Role: FOLLOWER
I20260620 14:52:35.616376 21083 consensus_queue.cc:260] T 00000000000000000000000000000000 P 0b13ed10a2614be6bd0f4d539303a89f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0b13ed10a2614be6bd0f4d539303a89f" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 46687 } }
I20260620 14:52:35.616623 21083 raft_consensus.cc:388] T 00000000000000000000000000000000 P 0b13ed10a2614be6bd0f4d539303a89f [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 14:52:35.616734 21083 raft_consensus.cc:482] T 00000000000000000000000000000000 P 0b13ed10a2614be6bd0f4d539303a89f [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 14:52:35.616864 21083 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 0b13ed10a2614be6bd0f4d539303a89f [term 0 FOLLOWER]: Advancing to term 1
I20260620 14:52:35.619148 21083 raft_consensus.cc:504] T 00000000000000000000000000000000 P 0b13ed10a2614be6bd0f4d539303a89f [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0b13ed10a2614be6bd0f4d539303a89f" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 46687 } }
I20260620 14:52:35.619791 21083 leader_election.cc:302] T 00000000000000000000000000000000 P 0b13ed10a2614be6bd0f4d539303a89f [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 0b13ed10a2614be6bd0f4d539303a89f; no voters:
I20260620 14:52:35.620486 21083 leader_election.cc:288] T 00000000000000000000000000000000 P 0b13ed10a2614be6bd0f4d539303a89f [CANDIDATE]: Term 1 election: Requested vote from peers
I20260620 14:52:35.620666 21093 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 0b13ed10a2614be6bd0f4d539303a89f [term 1 FOLLOWER]: Leader election won for term 1
I20260620 14:52:35.622012 21093 raft_consensus.cc:686] T 00000000000000000000000000000000 P 0b13ed10a2614be6bd0f4d539303a89f [term 1 LEADER]: Becoming Leader. State: Replica: 0b13ed10a2614be6bd0f4d539303a89f, State: Running, Role: LEADER
I20260620 14:52:35.622602 21093 consensus_queue.cc:237] T 00000000000000000000000000000000 P 0b13ed10a2614be6bd0f4d539303a89f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0b13ed10a2614be6bd0f4d539303a89f" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 46687 } }
I20260620 14:52:35.623009 21083 sys_catalog.cc:564] T 00000000000000000000000000000000 P 0b13ed10a2614be6bd0f4d539303a89f [sys.catalog]: configured and running, proceeding with master startup.
I20260620 14:52:35.626159 21094 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0b13ed10a2614be6bd0f4d539303a89f [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "0b13ed10a2614be6bd0f4d539303a89f" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0b13ed10a2614be6bd0f4d539303a89f" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 46687 } } }
I20260620 14:52:35.626137 21095 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0b13ed10a2614be6bd0f4d539303a89f [sys.catalog]: SysCatalogTable state changed. Reason: New leader 0b13ed10a2614be6bd0f4d539303a89f. Latest consensus state: current_term: 1 leader_uuid: "0b13ed10a2614be6bd0f4d539303a89f" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0b13ed10a2614be6bd0f4d539303a89f" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 46687 } } }
I20260620 14:52:35.627038 21094 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0b13ed10a2614be6bd0f4d539303a89f [sys.catalog]: This master's current role is: LEADER
I20260620 14:52:35.627044 21095 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0b13ed10a2614be6bd0f4d539303a89f [sys.catalog]: This master's current role is: LEADER
I20260620 14:52:35.629210 21102 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 14:52:35.634445 21102 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 14:52:35.644641 21102 catalog_manager.cc:1347] Generated new cluster ID: d04c993c83ad446d99b8d3b3e18bbbf0
I20260620 14:52:35.644793 21102 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 14:52:35.670115 21102 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 14:52:35.670734 21102 catalog_manager.cc:1504] Loading token signing keys...
I20260620 14:52:35.688778 21102 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 0b13ed10a2614be6bd0f4d539303a89f: Generated new TSK 0
I20260620 14:52:35.689781 21102 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260620 14:52:35.699533 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskUci3DU/build/asan/bin/kudu pid 21016
W20260620 14:52:35.700840 21078 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='slave'} at 127.0.0.1:42153
W20260620 14:52:35.701786 388 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskUci3DU/build/asan/bin/kudu pid 21016: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260620 14:52:35.701956 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskUci3DU/build/asan/bin/kudu with pid 21016
2026-06-20T14:52:35Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/2 (1951 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/3
Loading random data
Initializing database '/tmp/dist-test-taskUci3DU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1781966829091300-388-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 20 14:52:35 dist-test-slave-q9j6 krb5kdc[21114](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 20 14:52:35 dist-test-slave-q9j6 krb5kdc[21114](info): set up 2 sockets
Jun 20 14:52:35 dist-test-slave-q9j6 krb5kdc[21114](info): commencing operation
krb5kdc: starting...
W20260620 14:52:36.828519 388 mini_kdc.cc:121] Time spent starting KDC: real 1.090s user 0.000s sys 0.028s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskUci3DU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1781966829091300-388-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskUci3DU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1781966829091300-388-0/krb5kdc/test-user.keytab.
Jun 20 14:52:37 dist-test-slave-q9j6 krb5kdc[21114](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781967157, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-20T14:52:37Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T14:52:37Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskUci3DU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1781966829091300-388-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskUci3DU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1781966829091300-388-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260620 14:52:37.098939 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskUci3DU/build/asan/bin/kudu
/tmp/dist-test-taskUci3DU/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskUci3DU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1781966829091300-388-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskUci3DU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1781966829091300-388-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskUci3DU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1781966829091300-388-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskUci3DU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1781966829091300-388-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:41519
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskUci3DU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1781966829091300-388-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:36815
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:41519
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskUci3DU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1781966829091300-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskUci3DU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1781966829091300-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskUci3DU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1781966829091300-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 14:52:37.464169 21127 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 14:52:37.464478 21127 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 14:52:37.464692 21127 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 14:52:37.472208 21127 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 14:52:37.472343 21127 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 14:52:37.472466 21127 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 14:52:37.472576 21127 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 14:52:37.472671 21127 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 14:52:37.483011 21127 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:36815
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskUci3DU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1781966829091300-388-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskUci3DU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1781966829091300-388-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:41519
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskUci3DU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1781966829091300-388-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:41519
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskUci3DU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1781966829091300-388-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskUci3DU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1781966829091300-388-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 13:41:20 UTC on ccycloud.quasar-vdagoi.root.comops.site
build id 4457462
ASAN enabled
I20260620 14:52:37.484452 21127 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 14:52:37.485879 21127 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 14:52:37.498606 21135 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 14:52:37.498559 21132 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 14:52:37.498648 21133 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 14:52:37.499433 21127 server_base.cc:1034] running on GCE node
I20260620 14:52:37.500321 21127 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 14:52:37.501798 21127 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 14:52:37.503010 21127 hybrid_clock.cc:648] HybridClock initialized: now 1781967157502958 us; error 44 us; skew 500 ppm
I20260620 14:52:37.503407 21127 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 14:52:37 dist-test-slave-q9j6 krb5kdc[21114](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781967157, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 14:52:37.506817 21127 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260620 14:52:37.508592 21127 webserver.cc:460] Webserver started at http://127.0.0.1:34561/ using document root <none> and password file <none>
I20260620 14:52:37.509351 21127 fs_manager.cc:359] Metadata directory not provided
I20260620 14:52:37.509496 21127 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 14:52:37.509830 21127 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 14:52:37.513087 21127 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskUci3DU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1781966829091300-388-0/minicluster-data/master-0/data/instance:
uuid: "0f6c91a9bc7d41d3b3174746a2a9307b"
format_stamp: "Formatted at 2026-06-20 14:52:37 on dist-test-slave-q9j6"
server_key: "ebfa67a795d10cc323ac8134cc107cad"
server_key_iv: "dbf55440cd30c94bc281281584e41a90"
server_key_version: "encryptionkey@0"
I20260620 14:52:37.513908 21127 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskUci3DU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1781966829091300-388-0/minicluster-data/master-0/wal/instance:
uuid: "0f6c91a9bc7d41d3b3174746a2a9307b"
format_stamp: "Formatted at 2026-06-20 14:52:37 on dist-test-slave-q9j6"
server_key: "ebfa67a795d10cc323ac8134cc107cad"
server_key_iv: "dbf55440cd30c94bc281281584e41a90"
server_key_version: "encryptionkey@0"
I20260620 14:52:37.519565 21127 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.001s
I20260620 14:52:37.523339 21142 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 14:52:37.524878 21127 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.000s sys 0.003s
I20260620 14:52:37.525094 21127 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskUci3DU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1781966829091300-388-0/minicluster-data/master-0/data,/tmp/dist-test-taskUci3DU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1781966829091300-388-0/minicluster-data/master-0/wal
uuid: "0f6c91a9bc7d41d3b3174746a2a9307b"
format_stamp: "Formatted at 2026-06-20 14:52:37 on dist-test-slave-q9j6"
server_key: "ebfa67a795d10cc323ac8134cc107cad"
server_key_iv: "dbf55440cd30c94bc281281584e41a90"
server_key_version: "encryptionkey@0"
I20260620 14:52:37.525321 21127 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskUci3DU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1781966829091300-388-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskUci3DU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1781966829091300-388-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskUci3DU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1781966829091300-388-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 14:52:37.541882 21127 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 14:52:37.546124 21127 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 14:52:37.546447 21127 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 14:52:37.566066 21127 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:41519
I20260620 14:52:37.566089 21193 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:41519 every 8 connection(s)
I20260620 14:52:37.567577 21127 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskUci3DU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1781966829091300-388-0/minicluster-data/master-0/data/info.pb
I20260620 14:52:37.571767 21194 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 14:52:37.575882 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskUci3DU/build/asan/bin/kudu as pid 21127
I20260620 14:52:37.576098 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskUci3DU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1781966829091300-388-0/minicluster-data/master-0/wal/instance
I20260620 14:52:37.576643 388 external_mini_cluster.cc:1428] Setting key c1d04d8dbffb26e90986ab1ee63a5687
I20260620 14:52:37.585168 21194 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 0f6c91a9bc7d41d3b3174746a2a9307b: Bootstrap starting.
I20260620 14:52:37.588807 21194 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 0f6c91a9bc7d41d3b3174746a2a9307b: Neither blocks nor log segments found. Creating new log.
I20260620 14:52:37.590021 21194 log.cc:824] T 00000000000000000000000000000000 P 0f6c91a9bc7d41d3b3174746a2a9307b: Log is configured to *not* fsync() on all Append() calls
Jun 20 14:52:37 dist-test-slave-q9j6 krb5kdc[21114](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781967157, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.0.1@KRBTEST.COM
I20260620 14:52:37.593690 21194 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 0f6c91a9bc7d41d3b3174746a2a9307b: No bootstrap required, opened a new log
I20260620 14:52:37.602025 21197 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:52:37.578669 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53845 (local address 127.0.0.1:41519)
0620 14:52:37.579535 (+ 866us) server_negotiation.cc:206] Beginning negotiation
0620 14:52:37.579549 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0620 14:52:37.579646 (+ 97us) server_negotiation.cc:407] Connection header received
0620 14:52:37.581444 (+ 1798us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:52:37.581471 (+ 27us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:52:37.581980 (+ 509us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:52:37.582448 (+ 468us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:52:37.584062 (+ 1614us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:52:37.585510 (+ 1448us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:52:37.586922 (+ 1412us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:52:37.587738 (+ 816us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:52:37.587949 (+ 211us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:52:37.592697 (+ 4748us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 14:52:37.592713 (+ 16us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 14:52:37.592736 (+ 23us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 14:52:37.592795 (+ 59us) server_negotiation.cc:922] calling sasl_server_start()
0620 14:52:37.596941 (+ 4146us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:52:37.597742 (+ 801us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:52:37.597751 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:52:37.597761 (+ 10us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:52:37.597999 (+ 238us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:52:37.598390 (+ 391us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:52:37.598395 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:52:37.598399 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:52:37.599186 (+ 787us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 14:52:37.599506 (+ 320us) server_negotiation.cc:1035] Waiting for connection context
0620 14:52:37.599945 (+ 439us) server_negotiation.cc:299] Negotiation successful
0620 14:52:37.600463 (+ 518us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":633,"thread_start_us":396,"threads_started":1}
I20260620 14:52:37.604969 21194 raft_consensus.cc:348] T 00000000000000000000000000000000 P 0f6c91a9bc7d41d3b3174746a2a9307b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0f6c91a9bc7d41d3b3174746a2a9307b" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 41519 } }
I20260620 14:52:37.605477 21194 raft_consensus.cc:374] T 00000000000000000000000000000000 P 0f6c91a9bc7d41d3b3174746a2a9307b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 14:52:37.605616 21194 raft_consensus.cc:729] T 00000000000000000000000000000000 P 0f6c91a9bc7d41d3b3174746a2a9307b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0f6c91a9bc7d41d3b3174746a2a9307b, State: Initialized, Role: FOLLOWER
I20260620 14:52:37.606367 21194 consensus_queue.cc:260] T 00000000000000000000000000000000 P 0f6c91a9bc7d41d3b3174746a2a9307b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0f6c91a9bc7d41d3b3174746a2a9307b" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 41519 } }
I20260620 14:52:37.606513 388 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260620 14:52:37.606635 21194 raft_consensus.cc:388] T 00000000000000000000000000000000 P 0f6c91a9bc7d41d3b3174746a2a9307b [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 14:52:37.606789 21194 raft_consensus.cc:482] T 00000000000000000000000000000000 P 0f6c91a9bc7d41d3b3174746a2a9307b [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 14:52:37.606978 21194 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 0f6c91a9bc7d41d3b3174746a2a9307b [term 0 FOLLOWER]: Advancing to term 1
I20260620 14:52:37.609345 21194 raft_consensus.cc:504] T 00000000000000000000000000000000 P 0f6c91a9bc7d41d3b3174746a2a9307b [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0f6c91a9bc7d41d3b3174746a2a9307b" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 41519 } }
I20260620 14:52:37.609982 21194 leader_election.cc:302] T 00000000000000000000000000000000 P 0f6c91a9bc7d41d3b3174746a2a9307b [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 0f6c91a9bc7d41d3b3174746a2a9307b; no voters:
I20260620 14:52:37.610596 21194 leader_election.cc:288] T 00000000000000000000000000000000 P 0f6c91a9bc7d41d3b3174746a2a9307b [CANDIDATE]: Term 1 election: Requested vote from peers
I20260620 14:52:37.610749 21203 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 0f6c91a9bc7d41d3b3174746a2a9307b [term 1 FOLLOWER]: Leader election won for term 1
I20260620 14:52:37.611361 21203 raft_consensus.cc:686] T 00000000000000000000000000000000 P 0f6c91a9bc7d41d3b3174746a2a9307b [term 1 LEADER]: Becoming Leader. State: Replica: 0f6c91a9bc7d41d3b3174746a2a9307b, State: Running, Role: LEADER
I20260620 14:52:37.612155 21203 consensus_queue.cc:237] T 00000000000000000000000000000000 P 0f6c91a9bc7d41d3b3174746a2a9307b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0f6c91a9bc7d41d3b3174746a2a9307b" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 41519 } }
I20260620 14:52:37.613814 21194 sys_catalog.cc:564] T 00000000000000000000000000000000 P 0f6c91a9bc7d41d3b3174746a2a9307b [sys.catalog]: configured and running, proceeding with master startup.
I20260620 14:52:37.617834 21204 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0f6c91a9bc7d41d3b3174746a2a9307b [sys.catalog]: SysCatalogTable state changed. Reason: New leader 0f6c91a9bc7d41d3b3174746a2a9307b. Latest consensus state: current_term: 1 leader_uuid: "0f6c91a9bc7d41d3b3174746a2a9307b" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0f6c91a9bc7d41d3b3174746a2a9307b" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 41519 } } }
I20260620 14:52:37.618294 21204 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0f6c91a9bc7d41d3b3174746a2a9307b [sys.catalog]: This master's current role is: LEADER
I20260620 14:52:37.618854 21205 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0f6c91a9bc7d41d3b3174746a2a9307b [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "0f6c91a9bc7d41d3b3174746a2a9307b" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0f6c91a9bc7d41d3b3174746a2a9307b" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 41519 } } }
I20260620 14:52:37.619141 21205 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0f6c91a9bc7d41d3b3174746a2a9307b [sys.catalog]: This master's current role is: LEADER
I20260620 14:52:37.622246 21213 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 14:52:37.634727 21197 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:52:37.612123 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42033 (local address 127.0.0.1:41519)
0620 14:52:37.612309 (+ 186us) server_negotiation.cc:206] Beginning negotiation
0620 14:52:37.612315 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 14:52:37.613140 (+ 825us) server_negotiation.cc:407] Connection header received
0620 14:52:37.613333 (+ 193us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:52:37.613338 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:52:37.613417 (+ 79us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:52:37.613536 (+ 119us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:52:37.616038 (+ 2502us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:52:37.617457 (+ 1419us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:52:37.619548 (+ 2091us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:52:37.620486 (+ 938us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:52:37.620667 (+ 181us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:52:37.624719 (+ 4052us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 14:52:37.624730 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 14:52:37.624738 (+ 8us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 14:52:37.624802 (+ 64us) server_negotiation.cc:922] calling sasl_server_start()
0620 14:52:37.630024 (+ 5222us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:52:37.631165 (+ 1141us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:52:37.631175 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:52:37.631179 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:52:37.631385 (+ 206us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:52:37.631952 (+ 567us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:52:37.631958 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:52:37.631961 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:52:37.632501 (+ 540us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 14:52:37.632794 (+ 293us) server_negotiation.cc:1035] Waiting for connection context
0620 14:52:37.634031 (+ 1237us) server_negotiation.cc:299] Negotiation successful
0620 14:52:37.634320 (+ 289us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":62}
I20260620 14:52:37.634869 21213 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 14:52:37.646592 21213 catalog_manager.cc:1347] Generated new cluster ID: 78a4a144666f46ea8d70ee015b5067ca
I20260620 14:52:37.646739 21213 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 14:52:37.655824 21213 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 14:52:37.657768 21213 catalog_manager.cc:1504] Loading token signing keys...
I20260620 14:52:37.665266 21213 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 0f6c91a9bc7d41d3b3174746a2a9307b: Generated new TSK 0
I20260620 14:52:37.666131 21213 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260620 14:52:37.677209 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskUci3DU/build/asan/bin/kudu pid 21127
I20260620 14:52:37.678552 21189 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 14:52:37.769299 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskUci3DU/build/asan/bin/kudu with pid 21127
2026-06-20T14:52:37Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/3 (2069 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/4
Loading random data
Initializing database '/tmp/dist-test-taskUci3DU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1781966829091300-388-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 20 14:52:37 dist-test-slave-q9j6 krb5kdc[21226](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 20 14:52:37 dist-test-slave-q9j6 krb5kdc[21226](info): set up 2 sockets
Jun 20 14:52:37 dist-test-slave-q9j6 krb5kdc[21226](info): commencing operation
krb5kdc: starting...
W20260620 14:52:38.876128 388 mini_kdc.cc:121] Time spent starting KDC: real 1.068s user 0.002s sys 0.026s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskUci3DU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1781966829091300-388-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskUci3DU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1781966829091300-388-0/krb5kdc/test-user.keytab.
Jun 20 14:52:39 dist-test-slave-q9j6 krb5kdc[21226](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781967159, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-20T14:52:39Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T14:52:39Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskUci3DU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1781966829091300-388-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskUci3DU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1781966829091300-388-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260620 14:52:39.146588 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskUci3DU/build/asan/bin/kudu
/tmp/dist-test-taskUci3DU/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskUci3DU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1781966829091300-388-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskUci3DU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1781966829091300-388-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskUci3DU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1781966829091300-388-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskUci3DU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1781966829091300-388-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:36405
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskUci3DU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1781966829091300-388-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:37661
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:36405
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskUci3DU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1781966829091300-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskUci3DU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1781966829091300-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskUci3DU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1781966829091300-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 14:52:39.494745 21239 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 14:52:39.495134 21239 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 14:52:39.495304 21239 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 14:52:39.503458 21239 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 14:52:39.503661 21239 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 14:52:39.503755 21239 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 14:52:39.503851 21239 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 14:52:39.503942 21239 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 14:52:39.514345 21239 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:37661
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskUci3DU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1781966829091300-388-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskUci3DU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1781966829091300-388-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:36405
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskUci3DU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1781966829091300-388-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:36405
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskUci3DU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1781966829091300-388-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskUci3DU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1781966829091300-388-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 13:41:20 UTC on ccycloud.quasar-vdagoi.root.comops.site
build id 4457462
ASAN enabled
I20260620 14:52:39.515873 21239 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 14:52:39.517277 21239 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 14:52:39.529663 21244 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 14:52:39.529727 21247 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 14:52:39.529712 21245 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 14:52:39.530531 21239 server_base.cc:1034] running on GCE node
I20260620 14:52:39.531457 21239 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 14:52:39.533031 21239 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 14:52:39.534276 21239 hybrid_clock.cc:648] HybridClock initialized: now 1781967159534242 us; error 57 us; skew 500 ppm
I20260620 14:52:39.534667 21239 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 14:52:39 dist-test-slave-q9j6 krb5kdc[21226](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781967159, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 14:52:39.543288 21239 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260620 14:52:39.545192 21239 webserver.cc:460] Webserver started at http://127.0.0.1:34463/ using document root <none> and password file <none>
I20260620 14:52:39.545996 21239 fs_manager.cc:359] Metadata directory not provided
I20260620 14:52:39.546142 21239 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 14:52:39.546484 21239 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 14:52:39.549759 21239 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskUci3DU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1781966829091300-388-0/minicluster-data/master-0/data/instance:
uuid: "60e3d6bc152445d3ac11b88dcd5a2a34"
format_stamp: "Formatted at 2026-06-20 14:52:39 on dist-test-slave-q9j6"
server_key: "f15ee27e1bd3f63b71773c660c9ac5bf"
server_key_iv: "81202a7e129dd9ddb26b2447e1feed77"
server_key_version: "encryptionkey@0"
I20260620 14:52:39.550631 21239 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskUci3DU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1781966829091300-388-0/minicluster-data/master-0/wal/instance:
uuid: "60e3d6bc152445d3ac11b88dcd5a2a34"
format_stamp: "Formatted at 2026-06-20 14:52:39 on dist-test-slave-q9j6"
server_key: "f15ee27e1bd3f63b71773c660c9ac5bf"
server_key_iv: "81202a7e129dd9ddb26b2447e1feed77"
server_key_version: "encryptionkey@0"
I20260620 14:52:39.556506 21239 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.001s
I20260620 14:52:39.560436 21254 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 14:52:39.562016 21239 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260620 14:52:39.562237 21239 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskUci3DU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1781966829091300-388-0/minicluster-data/master-0/data,/tmp/dist-test-taskUci3DU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1781966829091300-388-0/minicluster-data/master-0/wal
uuid: "60e3d6bc152445d3ac11b88dcd5a2a34"
format_stamp: "Formatted at 2026-06-20 14:52:39 on dist-test-slave-q9j6"
server_key: "f15ee27e1bd3f63b71773c660c9ac5bf"
server_key_iv: "81202a7e129dd9ddb26b2447e1feed77"
server_key_version: "encryptionkey@0"
I20260620 14:52:39.562475 21239 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskUci3DU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1781966829091300-388-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskUci3DU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1781966829091300-388-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskUci3DU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1781966829091300-388-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 14:52:39.582939 21239 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 14:52:39.587652 21239 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 14:52:39.587996 21239 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 14:52:39.607309 21239 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:36405
I20260620 14:52:39.607345 21305 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:36405 every 8 connection(s)
I20260620 14:52:39.608907 21239 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskUci3DU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1781966829091300-388-0/minicluster-data/master-0/data/info.pb
I20260620 14:52:39.613044 21306 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 14:52:39.614326 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskUci3DU/build/asan/bin/kudu as pid 21239
I20260620 14:52:39.614526 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskUci3DU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1781966829091300-388-0/minicluster-data/master-0/wal/instance
I20260620 14:52:39.615082 388 external_mini_cluster.cc:1428] Setting key db74c85431f9dc115b5d164c26b0ef95
I20260620 14:52:39.628500 21306 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 60e3d6bc152445d3ac11b88dcd5a2a34: Bootstrap starting.
I20260620 14:52:39.630165 21309 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:52:39.617104 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36211 (local address 127.0.0.1:36405)
0620 14:52:39.618013 (+ 909us) server_negotiation.cc:206] Beginning negotiation
0620 14:52:39.618027 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0620 14:52:39.618077 (+ 50us) server_negotiation.cc:407] Connection header received
0620 14:52:39.619968 (+ 1891us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:52:39.620002 (+ 34us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:52:39.620502 (+ 500us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:52:39.620888 (+ 386us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:52:39.621599 (+ 711us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:52:39.622942 (+ 1343us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:52:39.625140 (+ 2198us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:52:39.625944 (+ 804us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:52:39.626123 (+ 179us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:52:39.627486 (+ 1363us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 14:52:39.627496 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 14:52:39.627514 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 14:52:39.628177 (+ 663us) server_negotiation.cc:922] calling sasl_server_start()
0620 14:52:39.628262 (+ 85us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 14:52:39.628296 (+ 34us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 14:52:39.628394 (+ 98us) server_negotiation.cc:1035] Waiting for connection context
0620 14:52:39.628701 (+ 307us) server_negotiation.cc:299] Negotiation successful
0620 14:52:39.629096 (+ 395us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":664,"thread_start_us":425,"threads_started":1}
I20260620 14:52:39.632117 21306 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 60e3d6bc152445d3ac11b88dcd5a2a34: Neither blocks nor log segments found. Creating new log.
I20260620 14:52:39.633410 21306 log.cc:824] T 00000000000000000000000000000000 P 60e3d6bc152445d3ac11b88dcd5a2a34: Log is configured to *not* fsync() on all Append() calls
I20260620 14:52:39.634662 388 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260620 14:52:39.637346 21306 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 60e3d6bc152445d3ac11b88dcd5a2a34: No bootstrap required, opened a new log
I20260620 14:52:39.647580 21309 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:52:39.640239 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57521 (local address 127.0.0.1:36405)
0620 14:52:39.640481 (+ 242us) server_negotiation.cc:206] Beginning negotiation
0620 14:52:39.640500 (+ 19us) server_negotiation.cc:399] Waiting for connection header
0620 14:52:39.640940 (+ 440us) server_negotiation.cc:407] Connection header received
0620 14:52:39.641142 (+ 202us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:52:39.641147 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:52:39.641240 (+ 93us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:52:39.641371 (+ 131us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:52:39.641927 (+ 556us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:52:39.643058 (+ 1131us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:52:39.644750 (+ 1692us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:52:39.645481 (+ 731us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:52:39.645636 (+ 155us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:52:39.646787 (+ 1151us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 14:52:39.646794 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 14:52:39.646799 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 14:52:39.646824 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0620 14:52:39.646881 (+ 57us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 14:52:39.646907 (+ 26us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 14:52:39.647000 (+ 93us) server_negotiation.cc:1035] Waiting for connection context
0620 14:52:39.647162 (+ 162us) server_negotiation.cc:299] Negotiation successful
0620 14:52:39.647266 (+ 104us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":77}
I20260620 14:52:39.648130 21306 raft_consensus.cc:348] T 00000000000000000000000000000000 P 60e3d6bc152445d3ac11b88dcd5a2a34 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "60e3d6bc152445d3ac11b88dcd5a2a34" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 36405 } }
I20260620 14:52:39.648592 21306 raft_consensus.cc:374] T 00000000000000000000000000000000 P 60e3d6bc152445d3ac11b88dcd5a2a34 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 14:52:39.648727 21306 raft_consensus.cc:729] T 00000000000000000000000000000000 P 60e3d6bc152445d3ac11b88dcd5a2a34 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 60e3d6bc152445d3ac11b88dcd5a2a34, State: Initialized, Role: FOLLOWER
I20260620 14:52:39.649423 21306 consensus_queue.cc:260] T 00000000000000000000000000000000 P 60e3d6bc152445d3ac11b88dcd5a2a34 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "60e3d6bc152445d3ac11b88dcd5a2a34" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 36405 } }
I20260620 14:52:39.649708 21306 raft_consensus.cc:388] T 00000000000000000000000000000000 P 60e3d6bc152445d3ac11b88dcd5a2a34 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 14:52:39.649869 21306 raft_consensus.cc:482] T 00000000000000000000000000000000 P 60e3d6bc152445d3ac11b88dcd5a2a34 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 14:52:39.650053 21306 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 60e3d6bc152445d3ac11b88dcd5a2a34 [term 0 FOLLOWER]: Advancing to term 1
I20260620 14:52:39.652493 21306 raft_consensus.cc:504] T 00000000000000000000000000000000 P 60e3d6bc152445d3ac11b88dcd5a2a34 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "60e3d6bc152445d3ac11b88dcd5a2a34" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 36405 } }
I20260620 14:52:39.653163 21306 leader_election.cc:302] T 00000000000000000000000000000000 P 60e3d6bc152445d3ac11b88dcd5a2a34 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 60e3d6bc152445d3ac11b88dcd5a2a34; no voters:
I20260620 14:52:39.653894 21306 leader_election.cc:288] T 00000000000000000000000000000000 P 60e3d6bc152445d3ac11b88dcd5a2a34 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260620 14:52:39.654007 21316 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 60e3d6bc152445d3ac11b88dcd5a2a34 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 14:52:39.654734 21316 raft_consensus.cc:686] T 00000000000000000000000000000000 P 60e3d6bc152445d3ac11b88dcd5a2a34 [term 1 LEADER]: Becoming Leader. State: Replica: 60e3d6bc152445d3ac11b88dcd5a2a34, State: Running, Role: LEADER
I20260620 14:52:39.655421 21316 consensus_queue.cc:237] T 00000000000000000000000000000000 P 60e3d6bc152445d3ac11b88dcd5a2a34 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "60e3d6bc152445d3ac11b88dcd5a2a34" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 36405 } }
I20260620 14:52:39.656479 21306 sys_catalog.cc:564] T 00000000000000000000000000000000 P 60e3d6bc152445d3ac11b88dcd5a2a34 [sys.catalog]: configured and running, proceeding with master startup.
I20260620 14:52:39.658761 21317 sys_catalog.cc:455] T 00000000000000000000000000000000 P 60e3d6bc152445d3ac11b88dcd5a2a34 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "60e3d6bc152445d3ac11b88dcd5a2a34" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "60e3d6bc152445d3ac11b88dcd5a2a34" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 36405 } } }
I20260620 14:52:39.658707 21318 sys_catalog.cc:455] T 00000000000000000000000000000000 P 60e3d6bc152445d3ac11b88dcd5a2a34 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 60e3d6bc152445d3ac11b88dcd5a2a34. Latest consensus state: current_term: 1 leader_uuid: "60e3d6bc152445d3ac11b88dcd5a2a34" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "60e3d6bc152445d3ac11b88dcd5a2a34" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 36405 } } }
I20260620 14:52:39.659241 21317 sys_catalog.cc:458] T 00000000000000000000000000000000 P 60e3d6bc152445d3ac11b88dcd5a2a34 [sys.catalog]: This master's current role is: LEADER
I20260620 14:52:39.659274 21318 sys_catalog.cc:458] T 00000000000000000000000000000000 P 60e3d6bc152445d3ac11b88dcd5a2a34 [sys.catalog]: This master's current role is: LEADER
I20260620 14:52:39.661537 21321 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 14:52:39.668735 21321 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 14:52:39.678551 21321 catalog_manager.cc:1347] Generated new cluster ID: 91787f020efc4b8a96d12754b21fac45
I20260620 14:52:39.678689 21321 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 14:52:39.708659 21321 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 14:52:39.710515 21321 catalog_manager.cc:1504] Loading token signing keys...
I20260620 14:52:39.722894 21321 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 60e3d6bc152445d3ac11b88dcd5a2a34: Generated new TSK 0
I20260620 14:52:39.724247 21321 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260620 14:52:39.741597 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskUci3DU/build/asan/bin/kudu pid 21239
W20260620 14:52:39.742916 21301 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='slave'} at 127.0.0.1:36211
W20260620 14:52:39.743728 388 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskUci3DU/build/asan/bin/kudu pid 21239: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260620 14:52:39.743891 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskUci3DU/build/asan/bin/kudu with pid 21239
2026-06-20T14:52:39Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/4 (1973 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/5
Loading random data
Initializing database '/tmp/dist-test-taskUci3DU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1781966829091300-388-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 20 14:52:39 dist-test-slave-q9j6 krb5kdc[21337](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 20 14:52:39 dist-test-slave-q9j6 krb5kdc[21337](info): set up 2 sockets
Jun 20 14:52:39 dist-test-slave-q9j6 krb5kdc[21337](info): commencing operation
krb5kdc: starting...
W20260620 14:52:40.852115 388 mini_kdc.cc:121] Time spent starting KDC: real 1.071s user 0.000s sys 0.029s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskUci3DU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1781966829091300-388-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskUci3DU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1781966829091300-388-0/krb5kdc/test-user.keytab.
Jun 20 14:52:41 dist-test-slave-q9j6 krb5kdc[21337](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781967161, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-20T14:52:41Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T14:52:41Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskUci3DU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1781966829091300-388-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskUci3DU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1781966829091300-388-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260620 14:52:41.125038 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskUci3DU/build/asan/bin/kudu
/tmp/dist-test-taskUci3DU/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskUci3DU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1781966829091300-388-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskUci3DU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1781966829091300-388-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskUci3DU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1781966829091300-388-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskUci3DU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1781966829091300-388-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:42071
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskUci3DU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1781966829091300-388-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:44577
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:42071
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskUci3DU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1781966829091300-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskUci3DU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1781966829091300-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskUci3DU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1781966829091300-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 14:52:41.482272 21350 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 14:52:41.482688 21350 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 14:52:41.482859 21350 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 14:52:41.490492 21350 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 14:52:41.490702 21350 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 14:52:41.490820 21350 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 14:52:41.490937 21350 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 14:52:41.491022 21350 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 14:52:41.501658 21350 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:44577
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskUci3DU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1781966829091300-388-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskUci3DU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1781966829091300-388-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:42071
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskUci3DU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1781966829091300-388-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:42071
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskUci3DU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1781966829091300-388-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskUci3DU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1781966829091300-388-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 13:41:20 UTC on ccycloud.quasar-vdagoi.root.comops.site
build id 4457462
ASAN enabled
I20260620 14:52:41.503134 21350 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 14:52:41.504665 21350 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 14:52:41.517490 21358 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 14:52:41.517530 21355 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 14:52:41.517553 21356 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 14:52:41.518291 21350 server_base.cc:1034] running on GCE node
I20260620 14:52:41.519279 21350 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 14:52:41.520879 21350 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 14:52:41.522115 21350 hybrid_clock.cc:648] HybridClock initialized: now 1781967161522068 us; error 33 us; skew 500 ppm
I20260620 14:52:41.522526 21350 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 14:52:41 dist-test-slave-q9j6 krb5kdc[21337](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781967161, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 14:52:41.525933 21350 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260620 14:52:41.527729 21350 webserver.cc:460] Webserver started at http://127.0.0.1:35213/ using document root <none> and password file <none>
I20260620 14:52:41.528484 21350 fs_manager.cc:359] Metadata directory not provided
I20260620 14:52:41.528599 21350 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 14:52:41.528872 21350 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 14:52:41.532178 21350 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskUci3DU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1781966829091300-388-0/minicluster-data/master-0/data/instance:
uuid: "fd8552a586004ab19a52cbbd6bd053bd"
format_stamp: "Formatted at 2026-06-20 14:52:41 on dist-test-slave-q9j6"
server_key: "03c23e45c84e94e404de0a1acd307504"
server_key_iv: "cbaca2978b862d2ad500d8eabbe5ee21"
server_key_version: "encryptionkey@0"
I20260620 14:52:41.532985 21350 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskUci3DU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1781966829091300-388-0/minicluster-data/master-0/wal/instance:
uuid: "fd8552a586004ab19a52cbbd6bd053bd"
format_stamp: "Formatted at 2026-06-20 14:52:41 on dist-test-slave-q9j6"
server_key: "03c23e45c84e94e404de0a1acd307504"
server_key_iv: "cbaca2978b862d2ad500d8eabbe5ee21"
server_key_version: "encryptionkey@0"
I20260620 14:52:41.538776 21350 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.001s
I20260620 14:52:41.542739 21365 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 14:52:41.544416 21350 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260620 14:52:41.544643 21350 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskUci3DU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1781966829091300-388-0/minicluster-data/master-0/data,/tmp/dist-test-taskUci3DU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1781966829091300-388-0/minicluster-data/master-0/wal
uuid: "fd8552a586004ab19a52cbbd6bd053bd"
format_stamp: "Formatted at 2026-06-20 14:52:41 on dist-test-slave-q9j6"
server_key: "03c23e45c84e94e404de0a1acd307504"
server_key_iv: "cbaca2978b862d2ad500d8eabbe5ee21"
server_key_version: "encryptionkey@0"
I20260620 14:52:41.544886 21350 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskUci3DU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1781966829091300-388-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskUci3DU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1781966829091300-388-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskUci3DU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1781966829091300-388-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 14:52:41.551329 21350 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 14:52:41.556049 21350 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 14:52:41.556396 21350 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 14:52:41.576596 21350 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:42071
I20260620 14:52:41.576632 21416 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:42071 every 8 connection(s)
I20260620 14:52:41.578255 21350 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskUci3DU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1781966829091300-388-0/minicluster-data/master-0/data/info.pb
I20260620 14:52:41.581923 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskUci3DU/build/asan/bin/kudu as pid 21350
I20260620 14:52:41.582127 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskUci3DU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1781966829091300-388-0/minicluster-data/master-0/wal/instance
I20260620 14:52:41.582702 388 external_mini_cluster.cc:1428] Setting key 29e8146fe264bece2ef42030e71a5f2e
I20260620 14:52:41.583055 21417 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 14:52:41.592397 21420 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:52:41.584716 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60227 (local address 127.0.0.1:42071)
0620 14:52:41.585673 (+ 957us) server_negotiation.cc:206] Beginning negotiation
0620 14:52:41.585687 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0620 14:52:41.585734 (+ 47us) server_negotiation.cc:407] Connection header received
0620 14:52:41.588255 (+ 2521us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:52:41.588287 (+ 32us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:52:41.588754 (+ 467us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:52:41.589214 (+ 460us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:52:41.590481 (+ 1267us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 14:52:41.590488 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 14:52:41.590508 (+ 20us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 14:52:41.590527 (+ 19us) server_negotiation.cc:922] calling sasl_server_start()
0620 14:52:41.590600 (+ 73us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 14:52:41.590631 (+ 31us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 14:52:41.590827 (+ 196us) server_negotiation.cc:1035] Waiting for connection context
0620 14:52:41.590976 (+ 149us) server_negotiation.cc:299] Negotiation successful
0620 14:52:41.591259 (+ 283us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":678,"thread_start_us":424,"threads_started":1}
I20260620 14:52:41.596518 388 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260620 14:52:41.598170 21417 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P fd8552a586004ab19a52cbbd6bd053bd: Bootstrap starting.
I20260620 14:52:41.602142 21417 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P fd8552a586004ab19a52cbbd6bd053bd: Neither blocks nor log segments found. Creating new log.
I20260620 14:52:41.603309 21417 log.cc:824] T 00000000000000000000000000000000 P fd8552a586004ab19a52cbbd6bd053bd: Log is configured to *not* fsync() on all Append() calls
I20260620 14:52:41.603819 21420 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:52:41.601024 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34449 (local address 127.0.0.1:42071)
0620 14:52:41.601244 (+ 220us) server_negotiation.cc:206] Beginning negotiation
0620 14:52:41.601252 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0620 14:52:41.601683 (+ 431us) server_negotiation.cc:407] Connection header received
0620 14:52:41.601898 (+ 215us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:52:41.601904 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:52:41.601967 (+ 63us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:52:41.602088 (+ 121us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:52:41.603089 (+ 1001us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 14:52:41.603094 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 14:52:41.603099 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 14:52:41.603118 (+ 19us) server_negotiation.cc:922] calling sasl_server_start()
0620 14:52:41.603171 (+ 53us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 14:52:41.603189 (+ 18us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 14:52:41.603318 (+ 129us) server_negotiation.cc:1035] Waiting for connection context
0620 14:52:41.603411 (+ 93us) server_negotiation.cc:299] Negotiation successful
0620 14:52:41.603476 (+ 65us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":55}
I20260620 14:52:41.606973 21417 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P fd8552a586004ab19a52cbbd6bd053bd: No bootstrap required, opened a new log
I20260620 14:52:41.615466 21417 raft_consensus.cc:348] T 00000000000000000000000000000000 P fd8552a586004ab19a52cbbd6bd053bd [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fd8552a586004ab19a52cbbd6bd053bd" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 42071 } }
I20260620 14:52:41.616016 21417 raft_consensus.cc:374] T 00000000000000000000000000000000 P fd8552a586004ab19a52cbbd6bd053bd [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 14:52:41.616142 21417 raft_consensus.cc:729] T 00000000000000000000000000000000 P fd8552a586004ab19a52cbbd6bd053bd [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fd8552a586004ab19a52cbbd6bd053bd, State: Initialized, Role: FOLLOWER
I20260620 14:52:41.616782 21417 consensus_queue.cc:260] T 00000000000000000000000000000000 P fd8552a586004ab19a52cbbd6bd053bd [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fd8552a586004ab19a52cbbd6bd053bd" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 42071 } }
I20260620 14:52:41.617041 21417 raft_consensus.cc:388] T 00000000000000000000000000000000 P fd8552a586004ab19a52cbbd6bd053bd [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 14:52:41.617206 21417 raft_consensus.cc:482] T 00000000000000000000000000000000 P fd8552a586004ab19a52cbbd6bd053bd [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 14:52:41.617379 21417 raft_consensus.cc:3037] T 00000000000000000000000000000000 P fd8552a586004ab19a52cbbd6bd053bd [term 0 FOLLOWER]: Advancing to term 1
I20260620 14:52:41.619807 21417 raft_consensus.cc:504] T 00000000000000000000000000000000 P fd8552a586004ab19a52cbbd6bd053bd [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fd8552a586004ab19a52cbbd6bd053bd" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 42071 } }
I20260620 14:52:41.620467 21417 leader_election.cc:302] T 00000000000000000000000000000000 P fd8552a586004ab19a52cbbd6bd053bd [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: fd8552a586004ab19a52cbbd6bd053bd; no voters:
I20260620 14:52:41.621151 21417 leader_election.cc:288] T 00000000000000000000000000000000 P fd8552a586004ab19a52cbbd6bd053bd [CANDIDATE]: Term 1 election: Requested vote from peers
I20260620 14:52:41.621320 21427 raft_consensus.cc:2781] T 00000000000000000000000000000000 P fd8552a586004ab19a52cbbd6bd053bd [term 1 FOLLOWER]: Leader election won for term 1
I20260620 14:52:41.622080 21427 raft_consensus.cc:686] T 00000000000000000000000000000000 P fd8552a586004ab19a52cbbd6bd053bd [term 1 LEADER]: Becoming Leader. State: Replica: fd8552a586004ab19a52cbbd6bd053bd, State: Running, Role: LEADER
I20260620 14:52:41.622685 21427 consensus_queue.cc:237] T 00000000000000000000000000000000 P fd8552a586004ab19a52cbbd6bd053bd [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fd8552a586004ab19a52cbbd6bd053bd" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 42071 } }
I20260620 14:52:41.623481 21417 sys_catalog.cc:564] T 00000000000000000000000000000000 P fd8552a586004ab19a52cbbd6bd053bd [sys.catalog]: configured and running, proceeding with master startup.
I20260620 14:52:41.625734 21428 sys_catalog.cc:455] T 00000000000000000000000000000000 P fd8552a586004ab19a52cbbd6bd053bd [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "fd8552a586004ab19a52cbbd6bd053bd" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fd8552a586004ab19a52cbbd6bd053bd" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 42071 } } }
I20260620 14:52:41.626237 21428 sys_catalog.cc:458] T 00000000000000000000000000000000 P fd8552a586004ab19a52cbbd6bd053bd [sys.catalog]: This master's current role is: LEADER
I20260620 14:52:41.626840 21429 sys_catalog.cc:455] T 00000000000000000000000000000000 P fd8552a586004ab19a52cbbd6bd053bd [sys.catalog]: SysCatalogTable state changed. Reason: New leader fd8552a586004ab19a52cbbd6bd053bd. Latest consensus state: current_term: 1 leader_uuid: "fd8552a586004ab19a52cbbd6bd053bd" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fd8552a586004ab19a52cbbd6bd053bd" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 42071 } } }
I20260620 14:52:41.627192 21429 sys_catalog.cc:458] T 00000000000000000000000000000000 P fd8552a586004ab19a52cbbd6bd053bd [sys.catalog]: This master's current role is: LEADER
I20260620 14:52:41.628198 21432 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 14:52:41.634022 21432 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 14:52:41.644827 21432 catalog_manager.cc:1347] Generated new cluster ID: d12cad24dde64678937f8aac360689ae
I20260620 14:52:41.644999 21432 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 14:52:41.654034 21432 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 14:52:41.654495 21432 catalog_manager.cc:1504] Loading token signing keys...
I20260620 14:52:41.662678 21432 catalog_manager.cc:5868] T 00000000000000000000000000000000 P fd8552a586004ab19a52cbbd6bd053bd: Generated new TSK 0
I20260620 14:52:41.663690 21432 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260620 14:52:41.673166 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskUci3DU/build/asan/bin/kudu pid 21350
W20260620 14:52:41.674760 21412 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='slave'} at 127.0.0.1:60227
W20260620 14:52:41.675680 388 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskUci3DU/build/asan/bin/kudu pid 21350: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260620 14:52:41.675858 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskUci3DU/build/asan/bin/kudu with pid 21350
2026-06-20T14:52:41Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/5 (1931 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/6
Loading random data
Initializing database '/tmp/dist-test-taskUci3DU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1781966829091300-388-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 20 14:52:41 dist-test-slave-q9j6 krb5kdc[21448](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 20 14:52:41 dist-test-slave-q9j6 krb5kdc[21448](info): set up 2 sockets
Jun 20 14:52:41 dist-test-slave-q9j6 krb5kdc[21448](info): commencing operation
krb5kdc: starting...
W20260620 14:52:42.776213 388 mini_kdc.cc:121] Time spent starting KDC: real 1.063s user 0.002s sys 0.026s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskUci3DU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1781966829091300-388-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskUci3DU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1781966829091300-388-0/krb5kdc/test-user.keytab.
Jun 20 14:52:42 dist-test-slave-q9j6 krb5kdc[21448](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781967162, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-20T14:52:42Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T14:52:42Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskUci3DU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1781966829091300-388-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskUci3DU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1781966829091300-388-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260620 14:52:43.046674 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskUci3DU/build/asan/bin/kudu
/tmp/dist-test-taskUci3DU/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskUci3DU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1781966829091300-388-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskUci3DU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1781966829091300-388-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskUci3DU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1781966829091300-388-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskUci3DU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1781966829091300-388-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:35801
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskUci3DU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1781966829091300-388-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:44887
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:35801
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskUci3DU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1781966829091300-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskUci3DU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1781966829091300-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskUci3DU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1781966829091300-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 14:52:43.400491 21461 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 14:52:43.400933 21461 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 14:52:43.401108 21461 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 14:52:43.408740 21461 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 14:52:43.408963 21461 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 14:52:43.409093 21461 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 14:52:43.409205 21461 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 14:52:43.409302 21461 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 14:52:43.420356 21461 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:44887
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskUci3DU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1781966829091300-388-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskUci3DU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1781966829091300-388-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:35801
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskUci3DU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1781966829091300-388-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:35801
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskUci3DU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1781966829091300-388-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskUci3DU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1781966829091300-388-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 13:41:20 UTC on ccycloud.quasar-vdagoi.root.comops.site
build id 4457462
ASAN enabled
I20260620 14:52:43.421841 21461 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 14:52:43.423306 21461 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 14:52:43.436210 21469 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 14:52:43.436268 21466 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 14:52:43.436250 21467 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 14:52:43.436810 21461 server_base.cc:1034] running on GCE node
I20260620 14:52:43.437741 21461 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 14:52:43.439286 21461 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 14:52:43.440496 21461 hybrid_clock.cc:648] HybridClock initialized: now 1781967163440434 us; error 57 us; skew 500 ppm
I20260620 14:52:43.440917 21461 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 14:52:43 dist-test-slave-q9j6 krb5kdc[21448](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781967163, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 14:52:43.444448 21461 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260620 14:52:43.446077 21461 webserver.cc:460] Webserver started at http://127.0.0.1:38933/ using document root <none> and password file <none>
I20260620 14:52:43.446887 21461 fs_manager.cc:359] Metadata directory not provided
I20260620 14:52:43.447041 21461 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 14:52:43.447392 21461 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 14:52:43.450783 21461 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskUci3DU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1781966829091300-388-0/minicluster-data/master-0/data/instance:
uuid: "e82b49730df44210b5e2e73d67f86cd9"
format_stamp: "Formatted at 2026-06-20 14:52:43 on dist-test-slave-q9j6"
server_key: "4c64506167bf18b128a84f4d4c48da60"
server_key_iv: "4515dee988a5c26d524ba29251f279f6"
server_key_version: "encryptionkey@0"
I20260620 14:52:43.451838 21461 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskUci3DU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1781966829091300-388-0/minicluster-data/master-0/wal/instance:
uuid: "e82b49730df44210b5e2e73d67f86cd9"
format_stamp: "Formatted at 2026-06-20 14:52:43 on dist-test-slave-q9j6"
server_key: "4c64506167bf18b128a84f4d4c48da60"
server_key_iv: "4515dee988a5c26d524ba29251f279f6"
server_key_version: "encryptionkey@0"
I20260620 14:52:43.457901 21461 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.001s sys 0.004s
I20260620 14:52:43.461999 21476 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 14:52:43.463662 21461 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260620 14:52:43.463888 21461 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskUci3DU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1781966829091300-388-0/minicluster-data/master-0/data,/tmp/dist-test-taskUci3DU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1781966829091300-388-0/minicluster-data/master-0/wal
uuid: "e82b49730df44210b5e2e73d67f86cd9"
format_stamp: "Formatted at 2026-06-20 14:52:43 on dist-test-slave-q9j6"
server_key: "4c64506167bf18b128a84f4d4c48da60"
server_key_iv: "4515dee988a5c26d524ba29251f279f6"
server_key_version: "encryptionkey@0"
I20260620 14:52:43.464119 21461 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskUci3DU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1781966829091300-388-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskUci3DU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1781966829091300-388-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskUci3DU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1781966829091300-388-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 14:52:43.497529 21461 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 14:52:43.502012 21461 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 14:52:43.502350 21461 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 14:52:43.521945 21461 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:35801
I20260620 14:52:43.521970 21527 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:35801 every 8 connection(s)
I20260620 14:52:43.523504 21461 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskUci3DU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1781966829091300-388-0/minicluster-data/master-0/data/info.pb
I20260620 14:52:43.523780 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskUci3DU/build/asan/bin/kudu as pid 21461
I20260620 14:52:43.523977 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskUci3DU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1781966829091300-388-0/minicluster-data/master-0/wal/instance
I20260620 14:52:43.524420 388 external_mini_cluster.cc:1428] Setting key 664e7a4b4d95329b028265676662f04a
I20260620 14:52:43.528553 21528 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
Jun 20 14:52:43 dist-test-slave-q9j6 krb5kdc[21448](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781967162, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.0.1@KRBTEST.COM
I20260620 14:52:43.542886 21528 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e82b49730df44210b5e2e73d67f86cd9: Bootstrap starting.
I20260620 14:52:43.546475 21528 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P e82b49730df44210b5e2e73d67f86cd9: Neither blocks nor log segments found. Creating new log.
I20260620 14:52:43.547712 21528 log.cc:824] T 00000000000000000000000000000000 P e82b49730df44210b5e2e73d67f86cd9: Log is configured to *not* fsync() on all Append() calls
I20260620 14:52:43.549943 21531 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:52:43.526534 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.95:53215 (local address 127.0.0.1:35801)
0620 14:52:43.527374 (+ 840us) server_negotiation.cc:206] Beginning negotiation
0620 14:52:43.527388 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0620 14:52:43.527439 (+ 51us) server_negotiation.cc:407] Connection header received
0620 14:52:43.529308 (+ 1869us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:52:43.529333 (+ 25us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:52:43.529792 (+ 459us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:52:43.530161 (+ 369us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:52:43.532127 (+ 1966us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:52:43.533560 (+ 1433us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:52:43.535573 (+ 2013us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:52:43.536407 (+ 834us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:52:43.536569 (+ 162us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:52:43.541802 (+ 5233us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 14:52:43.541812 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 14:52:43.541830 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 14:52:43.541876 (+ 46us) server_negotiation.cc:922] calling sasl_server_start()
0620 14:52:43.545218 (+ 3342us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:52:43.546172 (+ 954us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:52:43.546181 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:52:43.546188 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:52:43.546371 (+ 183us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:52:43.546955 (+ 584us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:52:43.546960 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:52:43.546961 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:52:43.547562 (+ 601us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 14:52:43.547959 (+ 397us) server_negotiation.cc:1035] Waiting for connection context
0620 14:52:43.548543 (+ 584us) server_negotiation.cc:299] Negotiation successful
0620 14:52:43.548895 (+ 352us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":607,"thread_start_us":360,"threads_started":1}
I20260620 14:52:43.551116 21528 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e82b49730df44210b5e2e73d67f86cd9: No bootstrap required, opened a new log
I20260620 14:52:43.554319 388 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260620 14:52:43.560546 21528 raft_consensus.cc:348] T 00000000000000000000000000000000 P e82b49730df44210b5e2e73d67f86cd9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e82b49730df44210b5e2e73d67f86cd9" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 35801 } }
I20260620 14:52:43.561015 21528 raft_consensus.cc:374] T 00000000000000000000000000000000 P e82b49730df44210b5e2e73d67f86cd9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 14:52:43.561131 21528 raft_consensus.cc:729] T 00000000000000000000000000000000 P e82b49730df44210b5e2e73d67f86cd9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e82b49730df44210b5e2e73d67f86cd9, State: Initialized, Role: FOLLOWER
I20260620 14:52:43.561740 21528 consensus_queue.cc:260] T 00000000000000000000000000000000 P e82b49730df44210b5e2e73d67f86cd9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e82b49730df44210b5e2e73d67f86cd9" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 35801 } }
I20260620 14:52:43.562029 21528 raft_consensus.cc:388] T 00000000000000000000000000000000 P e82b49730df44210b5e2e73d67f86cd9 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 14:52:43.562192 21528 raft_consensus.cc:482] T 00000000000000000000000000000000 P e82b49730df44210b5e2e73d67f86cd9 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 14:52:43.562386 21528 raft_consensus.cc:3037] T 00000000000000000000000000000000 P e82b49730df44210b5e2e73d67f86cd9 [term 0 FOLLOWER]: Advancing to term 1
I20260620 14:52:43.564786 21528 raft_consensus.cc:504] T 00000000000000000000000000000000 P e82b49730df44210b5e2e73d67f86cd9 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e82b49730df44210b5e2e73d67f86cd9" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 35801 } }
I20260620 14:52:43.565364 21528 leader_election.cc:302] T 00000000000000000000000000000000 P e82b49730df44210b5e2e73d67f86cd9 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: e82b49730df44210b5e2e73d67f86cd9; no voters:
I20260620 14:52:43.565997 21528 leader_election.cc:288] T 00000000000000000000000000000000 P e82b49730df44210b5e2e73d67f86cd9 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260620 14:52:43.566162 21538 raft_consensus.cc:2781] T 00000000000000000000000000000000 P e82b49730df44210b5e2e73d67f86cd9 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 14:52:43.567301 21538 raft_consensus.cc:686] T 00000000000000000000000000000000 P e82b49730df44210b5e2e73d67f86cd9 [term 1 LEADER]: Becoming Leader. State: Replica: e82b49730df44210b5e2e73d67f86cd9, State: Running, Role: LEADER
I20260620 14:52:43.568058 21538 consensus_queue.cc:237] T 00000000000000000000000000000000 P e82b49730df44210b5e2e73d67f86cd9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e82b49730df44210b5e2e73d67f86cd9" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 35801 } }
I20260620 14:52:43.568524 21528 sys_catalog.cc:564] T 00000000000000000000000000000000 P e82b49730df44210b5e2e73d67f86cd9 [sys.catalog]: configured and running, proceeding with master startup.
I20260620 14:52:43.571206 21539 sys_catalog.cc:455] T 00000000000000000000000000000000 P e82b49730df44210b5e2e73d67f86cd9 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "e82b49730df44210b5e2e73d67f86cd9" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e82b49730df44210b5e2e73d67f86cd9" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 35801 } } }
I20260620 14:52:43.571692 21539 sys_catalog.cc:458] T 00000000000000000000000000000000 P e82b49730df44210b5e2e73d67f86cd9 [sys.catalog]: This master's current role is: LEADER
I20260620 14:52:43.574249 21540 sys_catalog.cc:455] T 00000000000000000000000000000000 P e82b49730df44210b5e2e73d67f86cd9 [sys.catalog]: SysCatalogTable state changed. Reason: New leader e82b49730df44210b5e2e73d67f86cd9. Latest consensus state: current_term: 1 leader_uuid: "e82b49730df44210b5e2e73d67f86cd9" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e82b49730df44210b5e2e73d67f86cd9" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 35801 } } }
I20260620 14:52:43.574644 21540 sys_catalog.cc:458] T 00000000000000000000000000000000 P e82b49730df44210b5e2e73d67f86cd9 [sys.catalog]: This master's current role is: LEADER
I20260620 14:52:43.574775 21543 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 14:52:43.579960 21531 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:52:43.559037 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.95:54567 (local address 127.0.0.1:35801)
0620 14:52:43.559252 (+ 215us) server_negotiation.cc:206] Beginning negotiation
0620 14:52:43.559260 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0620 14:52:43.559648 (+ 388us) server_negotiation.cc:407] Connection header received
0620 14:52:43.559866 (+ 218us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:52:43.559870 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:52:43.559937 (+ 67us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:52:43.560045 (+ 108us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:52:43.561698 (+ 1653us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:52:43.562686 (+ 988us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:52:43.564092 (+ 1406us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:52:43.564546 (+ 454us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:52:43.564690 (+ 144us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:52:43.568578 (+ 3888us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 14:52:43.568585 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 14:52:43.568591 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 14:52:43.568643 (+ 52us) server_negotiation.cc:922] calling sasl_server_start()
0620 14:52:43.573586 (+ 4943us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:52:43.575154 (+ 1568us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:52:43.575163 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:52:43.575166 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:52:43.575390 (+ 224us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:52:43.576235 (+ 845us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:52:43.576243 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:52:43.576249 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:52:43.576818 (+ 569us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 14:52:43.578777 (+ 1959us) server_negotiation.cc:1035] Waiting for connection context
0620 14:52:43.579048 (+ 271us) server_negotiation.cc:299] Negotiation successful
0620 14:52:43.579347 (+ 299us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":60}
I20260620 14:52:43.580749 21543 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 14:52:43.592212 21543 catalog_manager.cc:1347] Generated new cluster ID: e845ad89abea464eb6726d5ad4785fd7
I20260620 14:52:43.592404 21543 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 14:52:43.604296 21543 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 14:52:43.605768 21543 catalog_manager.cc:1504] Loading token signing keys...
I20260620 14:52:43.630662 21543 catalog_manager.cc:5868] T 00000000000000000000000000000000 P e82b49730df44210b5e2e73d67f86cd9: Generated new TSK 0
I20260620 14:52:43.631692 21543 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260620 14:52:43.646706 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskUci3DU/build/asan/bin/kudu pid 21461
I20260620 14:52:43.648072 21523 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 14:52:43.743309 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskUci3DU/build/asan/bin/kudu with pid 21461
W20260620 14:52:43.761835 21446 connection.cc:535] client connection to 127.0.0.1:35801 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:35801): Connection reset by peer (error 104)
2026-06-20T14:52:43Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/6 (2068 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/7
Loading random data
Initializing database '/tmp/dist-test-taskUci3DU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1781966829091300-388-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 20 14:52:43 dist-test-slave-q9j6 krb5kdc[21560](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 20 14:52:43 dist-test-slave-q9j6 krb5kdc[21560](info): set up 2 sockets
Jun 20 14:52:43 dist-test-slave-q9j6 krb5kdc[21560](info): commencing operation
krb5kdc: starting...
W20260620 14:52:44.860213 388 mini_kdc.cc:121] Time spent starting KDC: real 1.079s user 0.000s sys 0.028s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskUci3DU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1781966829091300-388-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskUci3DU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1781966829091300-388-0/krb5kdc/test-user.keytab.
Jun 20 14:52:45 dist-test-slave-q9j6 krb5kdc[21560](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781967165, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-20T14:52:45Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T14:52:45Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskUci3DU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1781966829091300-388-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskUci3DU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1781966829091300-388-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260620 14:52:45.131181 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskUci3DU/build/asan/bin/kudu
/tmp/dist-test-taskUci3DU/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskUci3DU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1781966829091300-388-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskUci3DU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1781966829091300-388-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskUci3DU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1781966829091300-388-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskUci3DU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1781966829091300-388-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:38383
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskUci3DU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1781966829091300-388-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:38963
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:38383
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskUci3DU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1781966829091300-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskUci3DU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1781966829091300-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskUci3DU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1781966829091300-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 14:52:45.485306 21573 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 14:52:45.485687 21573 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 14:52:45.485862 21573 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 14:52:45.493435 21573 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 14:52:45.493634 21573 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 14:52:45.493753 21573 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 14:52:45.493867 21573 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 14:52:45.493945 21573 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 14:52:45.504436 21573 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:38963
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskUci3DU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1781966829091300-388-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskUci3DU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1781966829091300-388-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:38383
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskUci3DU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1781966829091300-388-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:38383
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskUci3DU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1781966829091300-388-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskUci3DU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1781966829091300-388-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 13:41:20 UTC on ccycloud.quasar-vdagoi.root.comops.site
build id 4457462
ASAN enabled
I20260620 14:52:45.505905 21573 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 14:52:45.507360 21573 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 14:52:45.520516 21578 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 14:52:45.520495 21579 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 14:52:45.520463 21581 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 14:52:45.521303 21573 server_base.cc:1034] running on GCE node
I20260620 14:52:45.522173 21573 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 14:52:45.523805 21573 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 14:52:45.525045 21573 hybrid_clock.cc:648] HybridClock initialized: now 1781967165525000 us; error 37 us; skew 500 ppm
I20260620 14:52:45.525465 21573 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 14:52:45 dist-test-slave-q9j6 krb5kdc[21560](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781967165, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 14:52:45.532459 21573 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260620 14:52:45.534286 21573 webserver.cc:460] Webserver started at http://127.0.0.1:34391/ using document root <none> and password file <none>
I20260620 14:52:45.535104 21573 fs_manager.cc:359] Metadata directory not provided
I20260620 14:52:45.535265 21573 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 14:52:45.535658 21573 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 14:52:45.538861 21573 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskUci3DU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1781966829091300-388-0/minicluster-data/master-0/data/instance:
uuid: "9b3ebf6569c54c36a10707aad2e1960d"
format_stamp: "Formatted at 2026-06-20 14:52:45 on dist-test-slave-q9j6"
server_key: "0c1485c706785b8e45fdde2b64404610"
server_key_iv: "a4ac3847c18e6c279a5fc2e6c56d340c"
server_key_version: "encryptionkey@0"
I20260620 14:52:45.539794 21573 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskUci3DU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1781966829091300-388-0/minicluster-data/master-0/wal/instance:
uuid: "9b3ebf6569c54c36a10707aad2e1960d"
format_stamp: "Formatted at 2026-06-20 14:52:45 on dist-test-slave-q9j6"
server_key: "0c1485c706785b8e45fdde2b64404610"
server_key_iv: "a4ac3847c18e6c279a5fc2e6c56d340c"
server_key_version: "encryptionkey@0"
I20260620 14:52:45.545568 21573 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260620 14:52:45.549598 21588 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 14:52:45.551184 21573 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260620 14:52:45.551415 21573 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskUci3DU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1781966829091300-388-0/minicluster-data/master-0/data,/tmp/dist-test-taskUci3DU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1781966829091300-388-0/minicluster-data/master-0/wal
uuid: "9b3ebf6569c54c36a10707aad2e1960d"
format_stamp: "Formatted at 2026-06-20 14:52:45 on dist-test-slave-q9j6"
server_key: "0c1485c706785b8e45fdde2b64404610"
server_key_iv: "a4ac3847c18e6c279a5fc2e6c56d340c"
server_key_version: "encryptionkey@0"
I20260620 14:52:45.551673 21573 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskUci3DU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1781966829091300-388-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskUci3DU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1781966829091300-388-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskUci3DU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1781966829091300-388-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 14:52:45.571712 21573 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 14:52:45.576745 21573 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 14:52:45.577085 21573 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 14:52:45.596279 21573 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:38383
I20260620 14:52:45.596285 21639 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:38383 every 8 connection(s)
I20260620 14:52:45.597849 21573 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskUci3DU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1781966829091300-388-0/minicluster-data/master-0/data/info.pb
I20260620 14:52:45.598361 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskUci3DU/build/asan/bin/kudu as pid 21573
I20260620 14:52:45.598551 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskUci3DU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1781966829091300-388-0/minicluster-data/master-0/wal/instance
I20260620 14:52:45.599146 388 external_mini_cluster.cc:1428] Setting key 263eafed2c5271a46fd7f4014e6a6c3a
I20260620 14:52:45.602908 21640 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 14:52:45.614816 21640 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 9b3ebf6569c54c36a10707aad2e1960d: Bootstrap starting.
I20260620 14:52:45.619220 21640 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 9b3ebf6569c54c36a10707aad2e1960d: Neither blocks nor log segments found. Creating new log.
I20260620 14:52:45.620468 21640 log.cc:824] T 00000000000000000000000000000000 P 9b3ebf6569c54c36a10707aad2e1960d: Log is configured to *not* fsync() on all Append() calls
I20260620 14:52:45.624270 21640 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 9b3ebf6569c54c36a10707aad2e1960d: No bootstrap required, opened a new log
I20260620 14:52:45.626417 21643 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:52:45.601235 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.95:58533 (local address 127.0.0.1:38383)
0620 14:52:45.602307 (+ 1072us) server_negotiation.cc:206] Beginning negotiation
0620 14:52:45.602330 (+ 23us) server_negotiation.cc:399] Waiting for connection header
0620 14:52:45.612387 (+ 10057us) server_negotiation.cc:407] Connection header received
0620 14:52:45.614959 (+ 2572us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:52:45.614994 (+ 35us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:52:45.615591 (+ 597us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:52:45.616156 (+ 565us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:52:45.616757 (+ 601us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:52:45.618853 (+ 2096us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:52:45.620507 (+ 1654us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:52:45.621593 (+ 1086us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:52:45.621761 (+ 168us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:52:45.623225 (+ 1464us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 14:52:45.623239 (+ 14us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 14:52:45.623261 (+ 22us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 14:52:45.624077 (+ 816us) server_negotiation.cc:922] calling sasl_server_start()
0620 14:52:45.624189 (+ 112us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 14:52:45.624227 (+ 38us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 14:52:45.624423 (+ 196us) server_negotiation.cc:1035] Waiting for connection context
0620 14:52:45.624742 (+ 319us) server_negotiation.cc:299] Negotiation successful
0620 14:52:45.624980 (+ 238us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":728,"thread_start_us":411,"threads_started":1}
I20260620 14:52:45.631062 388 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260620 14:52:45.633847 21640 raft_consensus.cc:348] T 00000000000000000000000000000000 P 9b3ebf6569c54c36a10707aad2e1960d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9b3ebf6569c54c36a10707aad2e1960d" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 38383 } }
I20260620 14:52:45.634398 21640 raft_consensus.cc:374] T 00000000000000000000000000000000 P 9b3ebf6569c54c36a10707aad2e1960d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 14:52:45.634543 21640 raft_consensus.cc:729] T 00000000000000000000000000000000 P 9b3ebf6569c54c36a10707aad2e1960d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9b3ebf6569c54c36a10707aad2e1960d, State: Initialized, Role: FOLLOWER
I20260620 14:52:45.635216 21640 consensus_queue.cc:260] T 00000000000000000000000000000000 P 9b3ebf6569c54c36a10707aad2e1960d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9b3ebf6569c54c36a10707aad2e1960d" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 38383 } }
I20260620 14:52:45.635547 21640 raft_consensus.cc:388] T 00000000000000000000000000000000 P 9b3ebf6569c54c36a10707aad2e1960d [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 14:52:45.635775 21640 raft_consensus.cc:482] T 00000000000000000000000000000000 P 9b3ebf6569c54c36a10707aad2e1960d [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 14:52:45.635995 21640 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 9b3ebf6569c54c36a10707aad2e1960d [term 0 FOLLOWER]: Advancing to term 1
I20260620 14:52:45.638581 21640 raft_consensus.cc:504] T 00000000000000000000000000000000 P 9b3ebf6569c54c36a10707aad2e1960d [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9b3ebf6569c54c36a10707aad2e1960d" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 38383 } }
I20260620 14:52:45.639150 21640 leader_election.cc:302] T 00000000000000000000000000000000 P 9b3ebf6569c54c36a10707aad2e1960d [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 9b3ebf6569c54c36a10707aad2e1960d; no voters:
I20260620 14:52:45.639897 21640 leader_election.cc:288] T 00000000000000000000000000000000 P 9b3ebf6569c54c36a10707aad2e1960d [CANDIDATE]: Term 1 election: Requested vote from peers
I20260620 14:52:45.640012 21650 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 9b3ebf6569c54c36a10707aad2e1960d [term 1 FOLLOWER]: Leader election won for term 1
I20260620 14:52:45.642810 21640 sys_catalog.cc:564] T 00000000000000000000000000000000 P 9b3ebf6569c54c36a10707aad2e1960d [sys.catalog]: configured and running, proceeding with master startup.
I20260620 14:52:45.643165 21643 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:52:45.635836 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.95:37281 (local address 127.0.0.1:38383)
0620 14:52:45.636042 (+ 206us) server_negotiation.cc:206] Beginning negotiation
0620 14:52:45.636047 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 14:52:45.636729 (+ 682us) server_negotiation.cc:407] Connection header received
0620 14:52:45.636968 (+ 239us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:52:45.636974 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:52:45.637062 (+ 88us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:52:45.637177 (+ 115us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:52:45.637737 (+ 560us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:52:45.638531 (+ 794us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:52:45.640145 (+ 1614us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:52:45.640926 (+ 781us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:52:45.641671 (+ 745us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:52:45.642293 (+ 622us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 14:52:45.642297 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 14:52:45.642301 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 14:52:45.642321 (+ 20us) server_negotiation.cc:922] calling sasl_server_start()
0620 14:52:45.642381 (+ 60us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 14:52:45.642396 (+ 15us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 14:52:45.642517 (+ 121us) server_negotiation.cc:1035] Waiting for connection context
0620 14:52:45.642727 (+ 210us) server_negotiation.cc:299] Negotiation successful
0620 14:52:45.642783 (+ 56us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":65}
I20260620 14:52:45.647925 21650 raft_consensus.cc:686] T 00000000000000000000000000000000 P 9b3ebf6569c54c36a10707aad2e1960d [term 1 LEADER]: Becoming Leader. State: Replica: 9b3ebf6569c54c36a10707aad2e1960d, State: Running, Role: LEADER
I20260620 14:52:45.648782 21650 consensus_queue.cc:237] T 00000000000000000000000000000000 P 9b3ebf6569c54c36a10707aad2e1960d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9b3ebf6569c54c36a10707aad2e1960d" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 38383 } }
I20260620 14:52:45.651787 21651 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9b3ebf6569c54c36a10707aad2e1960d [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "9b3ebf6569c54c36a10707aad2e1960d" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9b3ebf6569c54c36a10707aad2e1960d" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 38383 } } }
I20260620 14:52:45.652220 21651 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9b3ebf6569c54c36a10707aad2e1960d [sys.catalog]: This master's current role is: LEADER
I20260620 14:52:45.652143 21652 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9b3ebf6569c54c36a10707aad2e1960d [sys.catalog]: SysCatalogTable state changed. Reason: New leader 9b3ebf6569c54c36a10707aad2e1960d. Latest consensus state: current_term: 1 leader_uuid: "9b3ebf6569c54c36a10707aad2e1960d" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9b3ebf6569c54c36a10707aad2e1960d" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 38383 } } }
I20260620 14:52:45.652522 21652 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9b3ebf6569c54c36a10707aad2e1960d [sys.catalog]: This master's current role is: LEADER
W20260620 14:52:45.658438 21664 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 9b3ebf6569c54c36a10707aad2e1960d: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260620 14:52:45.658571 21664 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260620 14:52:45.658725 21665 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 14:52:45.661077 21665 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 14:52:45.682487 21665 catalog_manager.cc:1347] Generated new cluster ID: 9d5d5b9fbe9a4a9ba7907f842bfd8c34
I20260620 14:52:45.682682 21665 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 14:52:45.751279 21665 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 14:52:45.752750 21665 catalog_manager.cc:1504] Loading token signing keys...
I20260620 14:52:45.760430 21665 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 9b3ebf6569c54c36a10707aad2e1960d: Generated new TSK 0
I20260620 14:52:45.761644 21665 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260620 14:52:45.783154 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskUci3DU/build/asan/bin/kudu pid 21573
W20260620 14:52:45.784413 21635 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='slave'} at 10.240.0.95:58533
W20260620 14:52:45.785195 388 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskUci3DU/build/asan/bin/kudu pid 21573: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260620 14:52:45.785321 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskUci3DU/build/asan/bin/kudu with pid 21573
W20260620 14:52:45.803117 21558 connection.cc:535] client connection to 127.0.0.1:38383 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:38383): Connection reset by peer (error 104)
2026-06-20T14:52:45Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/7 (2041 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/8
Loading random data
Initializing database '/tmp/dist-test-taskUci3DU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1781966829091300-388-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 20 14:52:45 dist-test-slave-q9j6 krb5kdc[21671](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 20 14:52:45 dist-test-slave-q9j6 krb5kdc[21671](info): set up 2 sockets
Jun 20 14:52:45 dist-test-slave-q9j6 krb5kdc[21671](info): commencing operation
krb5kdc: starting...
W20260620 14:52:46.896142 388 mini_kdc.cc:121] Time spent starting KDC: real 1.073s user 0.002s sys 0.026s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskUci3DU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1781966829091300-388-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskUci3DU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1781966829091300-388-0/krb5kdc/test-user.keytab.
Jun 20 14:52:47 dist-test-slave-q9j6 krb5kdc[21671](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781967167, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-20T14:52:47Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T14:52:47Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskUci3DU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1781966829091300-388-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskUci3DU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1781966829091300-388-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260620 14:52:47.167372 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskUci3DU/build/asan/bin/kudu
/tmp/dist-test-taskUci3DU/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskUci3DU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1781966829091300-388-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskUci3DU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1781966829091300-388-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskUci3DU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1781966829091300-388-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskUci3DU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1781966829091300-388-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:45371
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskUci3DU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1781966829091300-388-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:41839
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:45371
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskUci3DU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1781966829091300-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskUci3DU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1781966829091300-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskUci3DU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1781966829091300-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 14:52:47.534154 21684 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 14:52:47.534473 21684 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 14:52:47.534567 21684 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 14:52:47.542104 21684 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 14:52:47.542248 21684 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 14:52:47.542357 21684 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 14:52:47.542455 21684 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 14:52:47.542534 21684 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 14:52:47.553061 21684 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:41839
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskUci3DU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1781966829091300-388-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskUci3DU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1781966829091300-388-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:45371
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskUci3DU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1781966829091300-388-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:45371
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskUci3DU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1781966829091300-388-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskUci3DU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1781966829091300-388-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 13:41:20 UTC on ccycloud.quasar-vdagoi.root.comops.site
build id 4457462
ASAN enabled
I20260620 14:52:47.554502 21684 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 14:52:47.555910 21684 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 14:52:47.568622 21689 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 14:52:47.568728 21690 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 14:52:47.568720 21692 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 14:52:47.570178 21684 server_base.cc:1034] running on GCE node
I20260620 14:52:47.570963 21684 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 14:52:47.572540 21684 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 14:52:47.573798 21684 hybrid_clock.cc:648] HybridClock initialized: now 1781967167573735 us; error 42 us; skew 500 ppm
I20260620 14:52:47.574167 21684 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 14:52:47 dist-test-slave-q9j6 krb5kdc[21671](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781967167, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 14:52:47.577502 21684 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260620 14:52:47.579180 21684 webserver.cc:460] Webserver started at http://127.0.0.1:34687/ using document root <none> and password file <none>
I20260620 14:52:47.579972 21684 fs_manager.cc:359] Metadata directory not provided
I20260620 14:52:47.580083 21684 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 14:52:47.580367 21684 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 14:52:47.583782 21684 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskUci3DU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1781966829091300-388-0/minicluster-data/master-0/data/instance:
uuid: "8ca09a3c1ea945a49efae0635f7ac779"
format_stamp: "Formatted at 2026-06-20 14:52:47 on dist-test-slave-q9j6"
server_key: "103527ff5c455961ecd6a126cd047380"
server_key_iv: "740faf6a46a218771a30069587180da5"
server_key_version: "encryptionkey@0"
I20260620 14:52:47.584606 21684 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskUci3DU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1781966829091300-388-0/minicluster-data/master-0/wal/instance:
uuid: "8ca09a3c1ea945a49efae0635f7ac779"
format_stamp: "Formatted at 2026-06-20 14:52:47 on dist-test-slave-q9j6"
server_key: "103527ff5c455961ecd6a126cd047380"
server_key_iv: "740faf6a46a218771a30069587180da5"
server_key_version: "encryptionkey@0"
I20260620 14:52:47.590647 21684 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.006s sys 0.000s
I20260620 14:52:47.594700 21699 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 14:52:47.596385 21684 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260620 14:52:47.596621 21684 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskUci3DU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1781966829091300-388-0/minicluster-data/master-0/data,/tmp/dist-test-taskUci3DU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1781966829091300-388-0/minicluster-data/master-0/wal
uuid: "8ca09a3c1ea945a49efae0635f7ac779"
format_stamp: "Formatted at 2026-06-20 14:52:47 on dist-test-slave-q9j6"
server_key: "103527ff5c455961ecd6a126cd047380"
server_key_iv: "740faf6a46a218771a30069587180da5"
server_key_version: "encryptionkey@0"
I20260620 14:52:47.596858 21684 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskUci3DU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1781966829091300-388-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskUci3DU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1781966829091300-388-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskUci3DU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1781966829091300-388-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 14:52:47.603368 21684 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 14:52:47.608418 21684 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 14:52:47.608696 21684 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 14:52:47.633494 21684 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:45371
I20260620 14:52:47.633538 21750 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:45371 every 8 connection(s)
I20260620 14:52:47.635635 21684 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskUci3DU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1781966829091300-388-0/minicluster-data/master-0/data/info.pb
I20260620 14:52:47.641584 21751 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 14:52:47.644577 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskUci3DU/build/asan/bin/kudu as pid 21684
I20260620 14:52:47.644767 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskUci3DU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1781966829091300-388-0/minicluster-data/master-0/wal/instance
I20260620 14:52:47.645190 388 external_mini_cluster.cc:1428] Setting key 3a1f0dd5766f734bc6fc8b0ce72e59aa
I20260620 14:52:47.653945 21754 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:52:47.647224 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.95:58231 (local address 127.0.0.1:45371)
0620 14:52:47.648131 (+ 907us) server_negotiation.cc:206] Beginning negotiation
0620 14:52:47.648148 (+ 17us) server_negotiation.cc:399] Waiting for connection header
0620 14:52:47.648194 (+ 46us) server_negotiation.cc:407] Connection header received
0620 14:52:47.650009 (+ 1815us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:52:47.650036 (+ 27us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:52:47.650444 (+ 408us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:52:47.650916 (+ 472us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:52:47.652134 (+ 1218us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 14:52:47.652141 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 14:52:47.652156 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 14:52:47.652171 (+ 15us) server_negotiation.cc:922] calling sasl_server_start()
0620 14:52:47.652231 (+ 60us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 14:52:47.652256 (+ 25us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 14:52:47.652528 (+ 272us) server_negotiation.cc:1035] Waiting for connection context
0620 14:52:47.652690 (+ 162us) server_negotiation.cc:299] Negotiation successful
0620 14:52:47.652919 (+ 229us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":649,"thread_start_us":418,"threads_started":1}
I20260620 14:52:47.655501 21751 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 8ca09a3c1ea945a49efae0635f7ac779: Bootstrap starting.
I20260620 14:52:47.658260 388 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260620 14:52:47.659266 21751 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 8ca09a3c1ea945a49efae0635f7ac779: Neither blocks nor log segments found. Creating new log.
I20260620 14:52:47.660496 21751 log.cc:824] T 00000000000000000000000000000000 P 8ca09a3c1ea945a49efae0635f7ac779: Log is configured to *not* fsync() on all Append() calls
I20260620 14:52:47.664168 21751 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 8ca09a3c1ea945a49efae0635f7ac779: No bootstrap required, opened a new log
I20260620 14:52:47.665838 21754 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:52:47.663035 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.95:46883 (local address 127.0.0.1:45371)
0620 14:52:47.663250 (+ 215us) server_negotiation.cc:206] Beginning negotiation
0620 14:52:47.663259 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0620 14:52:47.663734 (+ 475us) server_negotiation.cc:407] Connection header received
0620 14:52:47.663954 (+ 220us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:52:47.663966 (+ 12us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:52:47.664017 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:52:47.664129 (+ 112us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:52:47.665128 (+ 999us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 14:52:47.665135 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 14:52:47.665144 (+ 9us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 14:52:47.665160 (+ 16us) server_negotiation.cc:922] calling sasl_server_start()
0620 14:52:47.665201 (+ 41us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 14:52:47.665217 (+ 16us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 14:52:47.665343 (+ 126us) server_negotiation.cc:1035] Waiting for connection context
0620 14:52:47.665483 (+ 140us) server_negotiation.cc:299] Negotiation successful
0620 14:52:47.665569 (+ 86us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":55}
I20260620 14:52:47.672688 21751 raft_consensus.cc:348] T 00000000000000000000000000000000 P 8ca09a3c1ea945a49efae0635f7ac779 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8ca09a3c1ea945a49efae0635f7ac779" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 45371 } }
I20260620 14:52:47.673388 21751 raft_consensus.cc:374] T 00000000000000000000000000000000 P 8ca09a3c1ea945a49efae0635f7ac779 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 14:52:47.673514 21751 raft_consensus.cc:729] T 00000000000000000000000000000000 P 8ca09a3c1ea945a49efae0635f7ac779 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8ca09a3c1ea945a49efae0635f7ac779, State: Initialized, Role: FOLLOWER
I20260620 14:52:47.674285 21751 consensus_queue.cc:260] T 00000000000000000000000000000000 P 8ca09a3c1ea945a49efae0635f7ac779 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8ca09a3c1ea945a49efae0635f7ac779" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 45371 } }
I20260620 14:52:47.674569 21751 raft_consensus.cc:388] T 00000000000000000000000000000000 P 8ca09a3c1ea945a49efae0635f7ac779 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 14:52:47.674731 21751 raft_consensus.cc:482] T 00000000000000000000000000000000 P 8ca09a3c1ea945a49efae0635f7ac779 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 14:52:47.674942 21751 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 8ca09a3c1ea945a49efae0635f7ac779 [term 0 FOLLOWER]: Advancing to term 1
I20260620 14:52:47.677446 21751 raft_consensus.cc:504] T 00000000000000000000000000000000 P 8ca09a3c1ea945a49efae0635f7ac779 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8ca09a3c1ea945a49efae0635f7ac779" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 45371 } }
I20260620 14:52:47.678117 21751 leader_election.cc:302] T 00000000000000000000000000000000 P 8ca09a3c1ea945a49efae0635f7ac779 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 8ca09a3c1ea945a49efae0635f7ac779; no voters:
I20260620 14:52:47.678939 21751 leader_election.cc:288] T 00000000000000000000000000000000 P 8ca09a3c1ea945a49efae0635f7ac779 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260620 14:52:47.679086 21761 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 8ca09a3c1ea945a49efae0635f7ac779 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 14:52:47.680104 21761 raft_consensus.cc:686] T 00000000000000000000000000000000 P 8ca09a3c1ea945a49efae0635f7ac779 [term 1 LEADER]: Becoming Leader. State: Replica: 8ca09a3c1ea945a49efae0635f7ac779, State: Running, Role: LEADER
I20260620 14:52:47.680810 21761 consensus_queue.cc:237] T 00000000000000000000000000000000 P 8ca09a3c1ea945a49efae0635f7ac779 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8ca09a3c1ea945a49efae0635f7ac779" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 45371 } }
I20260620 14:52:47.681699 21751 sys_catalog.cc:564] T 00000000000000000000000000000000 P 8ca09a3c1ea945a49efae0635f7ac779 [sys.catalog]: configured and running, proceeding with master startup.
I20260620 14:52:47.684190 21762 sys_catalog.cc:455] T 00000000000000000000000000000000 P 8ca09a3c1ea945a49efae0635f7ac779 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "8ca09a3c1ea945a49efae0635f7ac779" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8ca09a3c1ea945a49efae0635f7ac779" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 45371 } } }
I20260620 14:52:47.684796 21762 sys_catalog.cc:458] T 00000000000000000000000000000000 P 8ca09a3c1ea945a49efae0635f7ac779 [sys.catalog]: This master's current role is: LEADER
I20260620 14:52:47.684376 21763 sys_catalog.cc:455] T 00000000000000000000000000000000 P 8ca09a3c1ea945a49efae0635f7ac779 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 8ca09a3c1ea945a49efae0635f7ac779. Latest consensus state: current_term: 1 leader_uuid: "8ca09a3c1ea945a49efae0635f7ac779" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8ca09a3c1ea945a49efae0635f7ac779" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 45371 } } }
I20260620 14:52:47.685024 21763 sys_catalog.cc:458] T 00000000000000000000000000000000 P 8ca09a3c1ea945a49efae0635f7ac779 [sys.catalog]: This master's current role is: LEADER
I20260620 14:52:47.687757 21767 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 14:52:47.692982 21767 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 14:52:47.703691 21767 catalog_manager.cc:1347] Generated new cluster ID: 784e60b853d84116a9b6625b4483f501
I20260620 14:52:47.703836 21767 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 14:52:47.718516 21767 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 14:52:47.719017 21767 catalog_manager.cc:1504] Loading token signing keys...
I20260620 14:52:47.733866 21767 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 8ca09a3c1ea945a49efae0635f7ac779: Generated new TSK 0
I20260620 14:52:47.735054 21767 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260620 14:52:47.743878 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskUci3DU/build/asan/bin/kudu pid 21684
W20260620 14:52:47.745280 21746 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='slave'} at 10.240.0.95:58231
W20260620 14:52:47.746053 388 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskUci3DU/build/asan/bin/kudu pid 21684: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260620 14:52:47.746201 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskUci3DU/build/asan/bin/kudu with pid 21684
2026-06-20T14:52:47Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/8 (1960 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/9
Loading random data
Initializing database '/tmp/dist-test-taskUci3DU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1781966829091300-388-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 20 14:52:47 dist-test-slave-q9j6 krb5kdc[21782](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 20 14:52:47 dist-test-slave-q9j6 krb5kdc[21782](info): set up 2 sockets
Jun 20 14:52:47 dist-test-slave-q9j6 krb5kdc[21782](info): commencing operation
krb5kdc: starting...
W20260620 14:52:48.852051 388 mini_kdc.cc:121] Time spent starting KDC: real 1.069s user 0.002s sys 0.026s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskUci3DU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1781966829091300-388-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskUci3DU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1781966829091300-388-0/krb5kdc/test-user.keytab.
Jun 20 14:52:49 dist-test-slave-q9j6 krb5kdc[21782](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781967169, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-20T14:52:49Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T14:52:49Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskUci3DU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1781966829091300-388-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskUci3DU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1781966829091300-388-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260620 14:52:49.123653 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskUci3DU/build/asan/bin/kudu
/tmp/dist-test-taskUci3DU/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskUci3DU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1781966829091300-388-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskUci3DU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1781966829091300-388-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskUci3DU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1781966829091300-388-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskUci3DU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1781966829091300-388-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:33285
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskUci3DU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1781966829091300-388-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:34853
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:33285
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskUci3DU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1781966829091300-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskUci3DU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1781966829091300-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskUci3DU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1781966829091300-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 14:52:49.480763 21795 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 14:52:49.481187 21795 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 14:52:49.481359 21795 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 14:52:49.489110 21795 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 14:52:49.489305 21795 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 14:52:49.489434 21795 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 14:52:49.489544 21795 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 14:52:49.489624 21795 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 14:52:49.500043 21795 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:34853
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskUci3DU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1781966829091300-388-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskUci3DU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1781966829091300-388-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:33285
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskUci3DU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1781966829091300-388-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:33285
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskUci3DU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1781966829091300-388-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskUci3DU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1781966829091300-388-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 13:41:20 UTC on ccycloud.quasar-vdagoi.root.comops.site
build id 4457462
ASAN enabled
I20260620 14:52:49.501643 21795 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 14:52:49.503134 21795 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 14:52:49.516264 21800 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 14:52:49.516151 21801 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 14:52:49.516199 21803 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 14:52:49.517275 21795 server_base.cc:1034] running on GCE node
I20260620 14:52:49.518249 21795 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 14:52:49.520037 21795 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 14:52:49.521318 21795 hybrid_clock.cc:648] HybridClock initialized: now 1781967169521266 us; error 51 us; skew 500 ppm
I20260620 14:52:49.521741 21795 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 14:52:49 dist-test-slave-q9j6 krb5kdc[21782](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781967169, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 14:52:49.525233 21795 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260620 14:52:49.526893 21795 webserver.cc:460] Webserver started at http://127.0.0.1:44725/ using document root <none> and password file <none>
I20260620 14:52:49.527720 21795 fs_manager.cc:359] Metadata directory not provided
I20260620 14:52:49.527838 21795 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 14:52:49.528121 21795 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 14:52:49.531477 21795 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskUci3DU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1781966829091300-388-0/minicluster-data/master-0/data/instance:
uuid: "2d86d8a9487642aebd2a0108cf6c6b13"
format_stamp: "Formatted at 2026-06-20 14:52:49 on dist-test-slave-q9j6"
server_key: "c713fe080dadece2eda4ff2e75247884"
server_key_iv: "2635af6dce61248bd3a585afcfe364d3"
server_key_version: "encryptionkey@0"
I20260620 14:52:49.532407 21795 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskUci3DU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1781966829091300-388-0/minicluster-data/master-0/wal/instance:
uuid: "2d86d8a9487642aebd2a0108cf6c6b13"
format_stamp: "Formatted at 2026-06-20 14:52:49 on dist-test-slave-q9j6"
server_key: "c713fe080dadece2eda4ff2e75247884"
server_key_iv: "2635af6dce61248bd3a585afcfe364d3"
server_key_version: "encryptionkey@0"
I20260620 14:52:49.538341 21795 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.002s sys 0.003s
I20260620 14:52:49.542593 21810 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 14:52:49.544334 21795 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260620 14:52:49.544576 21795 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskUci3DU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1781966829091300-388-0/minicluster-data/master-0/data,/tmp/dist-test-taskUci3DU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1781966829091300-388-0/minicluster-data/master-0/wal
uuid: "2d86d8a9487642aebd2a0108cf6c6b13"
format_stamp: "Formatted at 2026-06-20 14:52:49 on dist-test-slave-q9j6"
server_key: "c713fe080dadece2eda4ff2e75247884"
server_key_iv: "2635af6dce61248bd3a585afcfe364d3"
server_key_version: "encryptionkey@0"
I20260620 14:52:49.544824 21795 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskUci3DU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1781966829091300-388-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskUci3DU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1781966829091300-388-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskUci3DU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1781966829091300-388-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 14:52:49.580833 21795 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 14:52:49.585517 21795 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 14:52:49.585852 21795 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 14:52:49.605929 21795 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:33285
I20260620 14:52:49.605948 21861 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:33285 every 8 connection(s)
I20260620 14:52:49.607520 21795 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskUci3DU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1781966829091300-388-0/minicluster-data/master-0/data/info.pb
I20260620 14:52:49.611558 21862 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 14:52:49.611578 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskUci3DU/build/asan/bin/kudu as pid 21795
I20260620 14:52:49.611833 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskUci3DU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1781966829091300-388-0/minicluster-data/master-0/wal/instance
I20260620 14:52:49.612293 388 external_mini_cluster.cc:1428] Setting key ed39d4222787c6c8c78ed5045f0e52ae
I20260620 14:52:49.628089 21862 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 2d86d8a9487642aebd2a0108cf6c6b13: Bootstrap starting.
Jun 20 14:52:49 dist-test-slave-q9j6 krb5kdc[21782](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781967169, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.0.1@KRBTEST.COM
I20260620 14:52:49.631527 21862 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 2d86d8a9487642aebd2a0108cf6c6b13: Neither blocks nor log segments found. Creating new log.
I20260620 14:52:49.632721 21862 log.cc:824] T 00000000000000000000000000000000 P 2d86d8a9487642aebd2a0108cf6c6b13: Log is configured to *not* fsync() on all Append() calls
I20260620 14:52:49.635942 21862 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 2d86d8a9487642aebd2a0108cf6c6b13: No bootstrap required, opened a new log
I20260620 14:52:49.637887 21865 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:52:49.614374 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.95:42779 (local address 127.0.0.1:33285)
0620 14:52:49.615310 (+ 936us) server_negotiation.cc:206] Beginning negotiation
0620 14:52:49.615323 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0620 14:52:49.615370 (+ 47us) server_negotiation.cc:407] Connection header received
0620 14:52:49.617169 (+ 1799us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:52:49.617196 (+ 27us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:52:49.617697 (+ 501us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:52:49.618112 (+ 415us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:52:49.620239 (+ 2127us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:52:49.621699 (+ 1460us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:52:49.623579 (+ 1880us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:52:49.624449 (+ 870us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:52:49.624618 (+ 169us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:52:49.629776 (+ 5158us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 14:52:49.629785 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 14:52:49.629802 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 14:52:49.629857 (+ 55us) server_negotiation.cc:922] calling sasl_server_start()
0620 14:52:49.633229 (+ 3372us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:52:49.634183 (+ 954us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:52:49.634190 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:52:49.634198 (+ 8us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:52:49.634390 (+ 192us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:52:49.634908 (+ 518us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:52:49.634913 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:52:49.634915 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:52:49.635484 (+ 569us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 14:52:49.635881 (+ 397us) server_negotiation.cc:1035] Waiting for connection context
0620 14:52:49.636444 (+ 563us) server_negotiation.cc:299] Negotiation successful
0620 14:52:49.636794 (+ 350us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":705,"thread_start_us":450,"threads_started":1}
I20260620 14:52:49.642443 388 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260620 14:52:49.645174 21862 raft_consensus.cc:348] T 00000000000000000000000000000000 P 2d86d8a9487642aebd2a0108cf6c6b13 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2d86d8a9487642aebd2a0108cf6c6b13" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 33285 } }
I20260620 14:52:49.645702 21862 raft_consensus.cc:374] T 00000000000000000000000000000000 P 2d86d8a9487642aebd2a0108cf6c6b13 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 14:52:49.645830 21862 raft_consensus.cc:729] T 00000000000000000000000000000000 P 2d86d8a9487642aebd2a0108cf6c6b13 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2d86d8a9487642aebd2a0108cf6c6b13, State: Initialized, Role: FOLLOWER
I20260620 14:52:49.646528 21862 consensus_queue.cc:260] T 00000000000000000000000000000000 P 2d86d8a9487642aebd2a0108cf6c6b13 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2d86d8a9487642aebd2a0108cf6c6b13" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 33285 } }
I20260620 14:52:49.646785 21862 raft_consensus.cc:388] T 00000000000000000000000000000000 P 2d86d8a9487642aebd2a0108cf6c6b13 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 14:52:49.646951 21862 raft_consensus.cc:482] T 00000000000000000000000000000000 P 2d86d8a9487642aebd2a0108cf6c6b13 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 14:52:49.647176 21862 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 2d86d8a9487642aebd2a0108cf6c6b13 [term 0 FOLLOWER]: Advancing to term 1
I20260620 14:52:49.649919 21862 raft_consensus.cc:504] T 00000000000000000000000000000000 P 2d86d8a9487642aebd2a0108cf6c6b13 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2d86d8a9487642aebd2a0108cf6c6b13" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 33285 } }
I20260620 14:52:49.650521 21862 leader_election.cc:302] T 00000000000000000000000000000000 P 2d86d8a9487642aebd2a0108cf6c6b13 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 2d86d8a9487642aebd2a0108cf6c6b13; no voters:
I20260620 14:52:49.651235 21862 leader_election.cc:288] T 00000000000000000000000000000000 P 2d86d8a9487642aebd2a0108cf6c6b13 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260620 14:52:49.651346 21872 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 2d86d8a9487642aebd2a0108cf6c6b13 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 14:52:49.652729 21872 raft_consensus.cc:686] T 00000000000000000000000000000000 P 2d86d8a9487642aebd2a0108cf6c6b13 [term 1 LEADER]: Becoming Leader. State: Replica: 2d86d8a9487642aebd2a0108cf6c6b13, State: Running, Role: LEADER
I20260620 14:52:49.653451 21872 consensus_queue.cc:237] T 00000000000000000000000000000000 P 2d86d8a9487642aebd2a0108cf6c6b13 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2d86d8a9487642aebd2a0108cf6c6b13" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 33285 } }
I20260620 14:52:49.654124 21862 sys_catalog.cc:564] T 00000000000000000000000000000000 P 2d86d8a9487642aebd2a0108cf6c6b13 [sys.catalog]: configured and running, proceeding with master startup.
I20260620 14:52:49.666182 21873 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2d86d8a9487642aebd2a0108cf6c6b13 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "2d86d8a9487642aebd2a0108cf6c6b13" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2d86d8a9487642aebd2a0108cf6c6b13" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 33285 } } }
I20260620 14:52:49.666620 21873 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2d86d8a9487642aebd2a0108cf6c6b13 [sys.catalog]: This master's current role is: LEADER
I20260620 14:52:49.666968 21865 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:52:49.647145 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.95:45817 (local address 127.0.0.1:33285)
0620 14:52:49.647408 (+ 263us) server_negotiation.cc:206] Beginning negotiation
0620 14:52:49.647418 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0620 14:52:49.650214 (+ 2796us) server_negotiation.cc:407] Connection header received
0620 14:52:49.650431 (+ 217us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:52:49.650437 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:52:49.650522 (+ 85us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:52:49.650633 (+ 111us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:52:49.652501 (+ 1868us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:52:49.653753 (+ 1252us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:52:49.655414 (+ 1661us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:52:49.656242 (+ 828us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:52:49.656396 (+ 154us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:52:49.659052 (+ 2656us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 14:52:49.659059 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 14:52:49.659062 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 14:52:49.659102 (+ 40us) server_negotiation.cc:922] calling sasl_server_start()
0620 14:52:49.662180 (+ 3078us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:52:49.663270 (+ 1090us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:52:49.663278 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:52:49.663284 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:52:49.663491 (+ 207us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:52:49.664149 (+ 658us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:52:49.664154 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:52:49.664157 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:52:49.664608 (+ 451us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 14:52:49.664970 (+ 362us) server_negotiation.cc:1035] Waiting for connection context
0620 14:52:49.666302 (+ 1332us) server_negotiation.cc:299] Negotiation successful
0620 14:52:49.666526 (+ 224us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":83}
I20260620 14:52:49.670075 21874 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2d86d8a9487642aebd2a0108cf6c6b13 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 2d86d8a9487642aebd2a0108cf6c6b13. Latest consensus state: current_term: 1 leader_uuid: "2d86d8a9487642aebd2a0108cf6c6b13" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2d86d8a9487642aebd2a0108cf6c6b13" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 33285 } } }
I20260620 14:52:49.670485 21874 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2d86d8a9487642aebd2a0108cf6c6b13 [sys.catalog]: This master's current role is: LEADER
I20260620 14:52:49.671757 21885 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 14:52:49.676401 21885 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 14:52:49.686654 21885 catalog_manager.cc:1347] Generated new cluster ID: fc8ca4ae76784ba28a7fb476529cf60d
I20260620 14:52:49.686874 21885 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 14:52:49.713799 21885 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 14:52:49.715818 21885 catalog_manager.cc:1504] Loading token signing keys...
I20260620 14:52:49.724951 21885 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 2d86d8a9487642aebd2a0108cf6c6b13: Generated new TSK 0
I20260620 14:52:49.726150 21885 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260620 14:52:49.736738 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskUci3DU/build/asan/bin/kudu pid 21795
I20260620 14:52:49.738268 21857 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 14:52:49.836056 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskUci3DU/build/asan/bin/kudu with pid 21795
W20260620 14:52:49.854266 21780 connection.cc:535] client connection to 127.0.0.1:33285 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:33285): Connection reset by peer (error 104)
2026-06-20T14:52:49Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/9 (2089 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/10
Loading random data
Initializing database '/tmp/dist-test-taskUci3DU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1781966829091300-388-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 20 14:52:49 dist-test-slave-q9j6 krb5kdc[21895](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 20 14:52:49 dist-test-slave-q9j6 krb5kdc[21895](info): set up 2 sockets
Jun 20 14:52:49 dist-test-slave-q9j6 krb5kdc[21895](info): commencing operation
krb5kdc: starting...
W20260620 14:52:50.964085 388 mini_kdc.cc:121] Time spent starting KDC: real 1.090s user 0.004s sys 0.025s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskUci3DU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1781966829091300-388-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskUci3DU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1781966829091300-388-0/krb5kdc/test-user.keytab.
Jun 20 14:52:51 dist-test-slave-q9j6 krb5kdc[21895](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781967171, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-20T14:52:51Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T14:52:51Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskUci3DU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1781966829091300-388-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskUci3DU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1781966829091300-388-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260620 14:52:51.236824 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskUci3DU/build/asan/bin/kudu
/tmp/dist-test-taskUci3DU/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskUci3DU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1781966829091300-388-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskUci3DU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1781966829091300-388-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskUci3DU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1781966829091300-388-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskUci3DU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1781966829091300-388-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:39299
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskUci3DU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1781966829091300-388-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:44707
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:39299
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskUci3DU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1781966829091300-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskUci3DU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1781966829091300-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskUci3DU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1781966829091300-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 14:52:51.584614 21908 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 14:52:51.584924 21908 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 14:52:51.585089 21908 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 14:52:51.592796 21908 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 14:52:51.592922 21908 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 14:52:51.593012 21908 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 14:52:51.593118 21908 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 14:52:51.593214 21908 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 14:52:51.603541 21908 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:44707
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskUci3DU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1781966829091300-388-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskUci3DU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1781966829091300-388-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:39299
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskUci3DU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1781966829091300-388-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:39299
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskUci3DU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1781966829091300-388-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskUci3DU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1781966829091300-388-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 13:41:20 UTC on ccycloud.quasar-vdagoi.root.comops.site
build id 4457462
ASAN enabled
I20260620 14:52:51.604954 21908 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 14:52:51.606336 21908 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 14:52:51.618924 21913 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 14:52:51.619248 21916 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 14:52:51.618948 21914 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 14:52:51.619791 21908 server_base.cc:1034] running on GCE node
I20260620 14:52:51.620735 21908 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 14:52:51.622437 21908 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 14:52:51.623751 21908 hybrid_clock.cc:648] HybridClock initialized: now 1781967171623664 us; error 66 us; skew 500 ppm
I20260620 14:52:51.624151 21908 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 14:52:51 dist-test-slave-q9j6 krb5kdc[21895](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781967171, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 14:52:51.633364 21908 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260620 14:52:51.635068 21908 webserver.cc:460] Webserver started at http://127.0.0.1:40635/ using document root <none> and password file <none>
I20260620 14:52:51.635947 21908 fs_manager.cc:359] Metadata directory not provided
I20260620 14:52:51.636070 21908 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 14:52:51.636370 21908 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 14:52:51.639762 21908 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskUci3DU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1781966829091300-388-0/minicluster-data/master-0/data/instance:
uuid: "0cc089208f0044fd8640504debbaccca"
format_stamp: "Formatted at 2026-06-20 14:52:51 on dist-test-slave-q9j6"
server_key: "086a52ec98bf40fb249018241da51ab2"
server_key_iv: "e18f17a69ee4057bbae26f4423f258ec"
server_key_version: "encryptionkey@0"
I20260620 14:52:51.640693 21908 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskUci3DU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1781966829091300-388-0/minicluster-data/master-0/wal/instance:
uuid: "0cc089208f0044fd8640504debbaccca"
format_stamp: "Formatted at 2026-06-20 14:52:51 on dist-test-slave-q9j6"
server_key: "086a52ec98bf40fb249018241da51ab2"
server_key_iv: "e18f17a69ee4057bbae26f4423f258ec"
server_key_version: "encryptionkey@0"
I20260620 14:52:51.646967 21908 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.006s sys 0.000s
I20260620 14:52:51.651175 21923 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 14:52:51.652971 21908 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.002s
I20260620 14:52:51.653156 21908 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskUci3DU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1781966829091300-388-0/minicluster-data/master-0/data,/tmp/dist-test-taskUci3DU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1781966829091300-388-0/minicluster-data/master-0/wal
uuid: "0cc089208f0044fd8640504debbaccca"
format_stamp: "Formatted at 2026-06-20 14:52:51 on dist-test-slave-q9j6"
server_key: "086a52ec98bf40fb249018241da51ab2"
server_key_iv: "e18f17a69ee4057bbae26f4423f258ec"
server_key_version: "encryptionkey@0"
I20260620 14:52:51.653350 21908 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskUci3DU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1781966829091300-388-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskUci3DU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1781966829091300-388-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskUci3DU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1781966829091300-388-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 14:52:51.682772 21908 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 14:52:51.687553 21908 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 14:52:51.687860 21908 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 14:52:51.708083 21908 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:39299
I20260620 14:52:51.708107 21974 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:39299 every 8 connection(s)
I20260620 14:52:51.709575 21908 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskUci3DU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1781966829091300-388-0/minicluster-data/master-0/data/info.pb
I20260620 14:52:51.713922 21975 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 14:52:51.714506 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskUci3DU/build/asan/bin/kudu as pid 21908
I20260620 14:52:51.714690 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskUci3DU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1781966829091300-388-0/minicluster-data/master-0/wal/instance
I20260620 14:52:51.715145 388 external_mini_cluster.cc:1428] Setting key 224078c6b2956ad10eba320e378f3098
I20260620 14:52:51.732158 21975 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 0cc089208f0044fd8640504debbaccca: Bootstrap starting.
I20260620 14:52:51.732316 21978 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:52:51.717194 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.95:33573 (local address 127.0.0.1:39299)
0620 14:52:51.718203 (+ 1009us) server_negotiation.cc:206] Beginning negotiation
0620 14:52:51.718222 (+ 19us) server_negotiation.cc:399] Waiting for connection header
0620 14:52:51.718282 (+ 60us) server_negotiation.cc:407] Connection header received
0620 14:52:51.720328 (+ 2046us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:52:51.720373 (+ 45us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:52:51.721117 (+ 744us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:52:51.721724 (+ 607us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:52:51.722415 (+ 691us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:52:51.724789 (+ 2374us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:52:51.726444 (+ 1655us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:52:51.727691 (+ 1247us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:52:51.727934 (+ 243us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:52:51.729212 (+ 1278us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 14:52:51.729223 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 14:52:51.729246 (+ 23us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 14:52:51.730062 (+ 816us) server_negotiation.cc:922] calling sasl_server_start()
0620 14:52:51.730167 (+ 105us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 14:52:51.730201 (+ 34us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 14:52:51.730363 (+ 162us) server_negotiation.cc:1035] Waiting for connection context
0620 14:52:51.730651 (+ 288us) server_negotiation.cc:299] Negotiation successful
0620 14:52:51.730862 (+ 211us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":681,"thread_start_us":428,"threads_started":1}
I20260620 14:52:51.735821 21975 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 0cc089208f0044fd8640504debbaccca: Neither blocks nor log segments found. Creating new log.
I20260620 14:52:51.736626 388 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260620 14:52:51.737166 21975 log.cc:824] T 00000000000000000000000000000000 P 0cc089208f0044fd8640504debbaccca: Log is configured to *not* fsync() on all Append() calls
I20260620 14:52:51.740464 21975 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 0cc089208f0044fd8640504debbaccca: No bootstrap required, opened a new log
I20260620 14:52:51.749143 21978 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:52:51.741887 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.95:37899 (local address 127.0.0.1:39299)
0620 14:52:51.742121 (+ 234us) server_negotiation.cc:206] Beginning negotiation
0620 14:52:51.742133 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0620 14:52:51.742638 (+ 505us) server_negotiation.cc:407] Connection header received
0620 14:52:51.742796 (+ 158us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:52:51.742802 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:52:51.742904 (+ 102us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:52:51.743024 (+ 120us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:52:51.743638 (+ 614us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:52:51.744818 (+ 1180us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:52:51.746168 (+ 1350us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:52:51.746898 (+ 730us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:52:51.747035 (+ 137us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:52:51.748251 (+ 1216us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 14:52:51.748258 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 14:52:51.748265 (+ 7us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 14:52:51.748303 (+ 38us) server_negotiation.cc:922] calling sasl_server_start()
0620 14:52:51.748364 (+ 61us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 14:52:51.748389 (+ 25us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 14:52:51.748504 (+ 115us) server_negotiation.cc:1035] Waiting for connection context
0620 14:52:51.748672 (+ 168us) server_negotiation.cc:299] Negotiation successful
0620 14:52:51.748747 (+ 75us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":50}
I20260620 14:52:51.750842 21975 raft_consensus.cc:348] T 00000000000000000000000000000000 P 0cc089208f0044fd8640504debbaccca [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0cc089208f0044fd8640504debbaccca" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 39299 } }
I20260620 14:52:51.751245 21975 raft_consensus.cc:374] T 00000000000000000000000000000000 P 0cc089208f0044fd8640504debbaccca [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 14:52:51.751323 21975 raft_consensus.cc:729] T 00000000000000000000000000000000 P 0cc089208f0044fd8640504debbaccca [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0cc089208f0044fd8640504debbaccca, State: Initialized, Role: FOLLOWER
I20260620 14:52:51.751945 21975 consensus_queue.cc:260] T 00000000000000000000000000000000 P 0cc089208f0044fd8640504debbaccca [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0cc089208f0044fd8640504debbaccca" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 39299 } }
I20260620 14:52:51.752168 21975 raft_consensus.cc:388] T 00000000000000000000000000000000 P 0cc089208f0044fd8640504debbaccca [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 14:52:51.752272 21975 raft_consensus.cc:482] T 00000000000000000000000000000000 P 0cc089208f0044fd8640504debbaccca [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 14:52:51.752401 21975 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 0cc089208f0044fd8640504debbaccca [term 0 FOLLOWER]: Advancing to term 1
I20260620 14:52:51.754621 21975 raft_consensus.cc:504] T 00000000000000000000000000000000 P 0cc089208f0044fd8640504debbaccca [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0cc089208f0044fd8640504debbaccca" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 39299 } }
I20260620 14:52:51.755141 21975 leader_election.cc:302] T 00000000000000000000000000000000 P 0cc089208f0044fd8640504debbaccca [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 0cc089208f0044fd8640504debbaccca; no voters:
I20260620 14:52:51.755846 21975 leader_election.cc:288] T 00000000000000000000000000000000 P 0cc089208f0044fd8640504debbaccca [CANDIDATE]: Term 1 election: Requested vote from peers
I20260620 14:52:51.756018 21985 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 0cc089208f0044fd8640504debbaccca [term 1 FOLLOWER]: Leader election won for term 1
I20260620 14:52:51.757244 21985 raft_consensus.cc:686] T 00000000000000000000000000000000 P 0cc089208f0044fd8640504debbaccca [term 1 LEADER]: Becoming Leader. State: Replica: 0cc089208f0044fd8640504debbaccca, State: Running, Role: LEADER
I20260620 14:52:51.758152 21975 sys_catalog.cc:564] T 00000000000000000000000000000000 P 0cc089208f0044fd8640504debbaccca [sys.catalog]: configured and running, proceeding with master startup.
I20260620 14:52:51.758190 21985 consensus_queue.cc:237] T 00000000000000000000000000000000 P 0cc089208f0044fd8640504debbaccca [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0cc089208f0044fd8640504debbaccca" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 39299 } }
I20260620 14:52:51.761445 21986 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0cc089208f0044fd8640504debbaccca [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "0cc089208f0044fd8640504debbaccca" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0cc089208f0044fd8640504debbaccca" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 39299 } } }
I20260620 14:52:51.761742 21987 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0cc089208f0044fd8640504debbaccca [sys.catalog]: SysCatalogTable state changed. Reason: New leader 0cc089208f0044fd8640504debbaccca. Latest consensus state: current_term: 1 leader_uuid: "0cc089208f0044fd8640504debbaccca" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0cc089208f0044fd8640504debbaccca" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 39299 } } }
I20260620 14:52:51.761916 21986 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0cc089208f0044fd8640504debbaccca [sys.catalog]: This master's current role is: LEADER
I20260620 14:52:51.761960 21987 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0cc089208f0044fd8640504debbaccca [sys.catalog]: This master's current role is: LEADER
I20260620 14:52:51.762777 21994 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 14:52:51.771265 21994 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 14:52:51.782354 21994 catalog_manager.cc:1347] Generated new cluster ID: f17a632270f543958bd590a5b623379d
I20260620 14:52:51.782511 21994 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 14:52:51.802578 21994 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 14:52:51.804072 21994 catalog_manager.cc:1504] Loading token signing keys...
I20260620 14:52:51.812074 21994 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 0cc089208f0044fd8640504debbaccca: Generated new TSK 0
I20260620 14:52:51.812983 21994 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260620 14:52:51.828918 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskUci3DU/build/asan/bin/kudu pid 21908
W20260620 14:52:51.830331 21970 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='slave'} at 10.240.0.95:33573
W20260620 14:52:51.831221 388 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskUci3DU/build/asan/bin/kudu pid 21908: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260620 14:52:51.831387 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskUci3DU/build/asan/bin/kudu with pid 21908
W20260620 14:52:51.849015 21893 connection.cc:535] client connection to 127.0.0.1:39299 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:39299): Connection reset by peer (error 104)
2026-06-20T14:52:51Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/10 (1996 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/11
Loading random data
Initializing database '/tmp/dist-test-taskUci3DU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1781966829091300-388-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 20 14:52:51 dist-test-slave-q9j6 krb5kdc[22006](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 20 14:52:51 dist-test-slave-q9j6 krb5kdc[22006](info): set up 2 sockets
Jun 20 14:52:51 dist-test-slave-q9j6 krb5kdc[22006](info): commencing operation
krb5kdc: starting...
W20260620 14:52:52.944154 388 mini_kdc.cc:121] Time spent starting KDC: real 1.074s user 0.000s sys 0.030s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskUci3DU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1781966829091300-388-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskUci3DU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1781966829091300-388-0/krb5kdc/test-user.keytab.
Jun 20 14:52:53 dist-test-slave-q9j6 krb5kdc[22006](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781967173, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-20T14:52:53Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T14:52:53Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskUci3DU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1781966829091300-388-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskUci3DU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1781966829091300-388-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260620 14:52:53.216773 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskUci3DU/build/asan/bin/kudu
/tmp/dist-test-taskUci3DU/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskUci3DU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1781966829091300-388-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskUci3DU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1781966829091300-388-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskUci3DU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1781966829091300-388-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskUci3DU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1781966829091300-388-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:40323
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskUci3DU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1781966829091300-388-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:39839
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:40323
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskUci3DU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1781966829091300-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskUci3DU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1781966829091300-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskUci3DU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1781966829091300-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 14:52:53.566540 22019 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 14:52:53.566932 22019 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 14:52:53.567099 22019 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 14:52:53.574723 22019 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 14:52:53.574903 22019 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 14:52:53.575023 22019 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 14:52:53.575133 22019 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 14:52:53.575212 22019 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 14:52:53.585670 22019 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:39839
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskUci3DU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1781966829091300-388-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskUci3DU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1781966829091300-388-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:40323
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskUci3DU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1781966829091300-388-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:40323
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskUci3DU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1781966829091300-388-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskUci3DU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1781966829091300-388-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 13:41:20 UTC on ccycloud.quasar-vdagoi.root.comops.site
build id 4457462
ASAN enabled
I20260620 14:52:53.587141 22019 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 14:52:53.588651 22019 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 14:52:53.601284 22024 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 14:52:53.601325 22025 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 14:52:53.601377 22027 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 14:52:53.602334 22019 server_base.cc:1034] running on GCE node
I20260620 14:52:53.603287 22019 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 14:52:53.604979 22019 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 14:52:53.606241 22019 hybrid_clock.cc:648] HybridClock initialized: now 1781967173606192 us; error 49 us; skew 500 ppm
I20260620 14:52:53.606644 22019 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 14:52:53 dist-test-slave-q9j6 krb5kdc[22006](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781967173, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 14:52:53.610008 22019 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260620 14:52:53.611819 22019 webserver.cc:460] Webserver started at http://127.0.0.1:41769/ using document root <none> and password file <none>
I20260620 14:52:53.612669 22019 fs_manager.cc:359] Metadata directory not provided
I20260620 14:52:53.612838 22019 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 14:52:53.613181 22019 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 14:52:53.616559 22019 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskUci3DU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1781966829091300-388-0/minicluster-data/master-0/data/instance:
uuid: "8939d190f17a4cbabb58a59eef2891d2"
format_stamp: "Formatted at 2026-06-20 14:52:53 on dist-test-slave-q9j6"
server_key: "81e79fe7162e9e128f8ce32a87331b98"
server_key_iv: "13b8d951d27f90d8b5af571d90d2d654"
server_key_version: "encryptionkey@0"
I20260620 14:52:53.617477 22019 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskUci3DU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1781966829091300-388-0/minicluster-data/master-0/wal/instance:
uuid: "8939d190f17a4cbabb58a59eef2891d2"
format_stamp: "Formatted at 2026-06-20 14:52:53 on dist-test-slave-q9j6"
server_key: "81e79fe7162e9e128f8ce32a87331b98"
server_key_iv: "13b8d951d27f90d8b5af571d90d2d654"
server_key_version: "encryptionkey@0"
I20260620 14:52:53.623366 22019 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.003s sys 0.004s
I20260620 14:52:53.627169 22034 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 14:52:53.628726 22019 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260620 14:52:53.628942 22019 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskUci3DU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1781966829091300-388-0/minicluster-data/master-0/data,/tmp/dist-test-taskUci3DU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1781966829091300-388-0/minicluster-data/master-0/wal
uuid: "8939d190f17a4cbabb58a59eef2891d2"
format_stamp: "Formatted at 2026-06-20 14:52:53 on dist-test-slave-q9j6"
server_key: "81e79fe7162e9e128f8ce32a87331b98"
server_key_iv: "13b8d951d27f90d8b5af571d90d2d654"
server_key_version: "encryptionkey@0"
I20260620 14:52:53.629173 22019 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskUci3DU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1781966829091300-388-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskUci3DU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1781966829091300-388-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskUci3DU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1781966829091300-388-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 14:52:53.635496 22019 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 14:52:53.640033 22019 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 14:52:53.640370 22019 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 14:52:53.660346 22019 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:40323
I20260620 14:52:53.660378 22085 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:40323 every 8 connection(s)
I20260620 14:52:53.662005 22019 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskUci3DU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1781966829091300-388-0/minicluster-data/master-0/data/info.pb
I20260620 14:52:53.663751 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskUci3DU/build/asan/bin/kudu as pid 22019
I20260620 14:52:53.663933 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskUci3DU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1781966829091300-388-0/minicluster-data/master-0/wal/instance
I20260620 14:52:53.664405 388 external_mini_cluster.cc:1428] Setting key abcdb5cd3c04b438a5a6c900ad1931b2
I20260620 14:52:53.666985 22086 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 14:52:53.672791 22089 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:52:53.666267 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.95:55895 (local address 127.0.0.1:40323)
0620 14:52:53.667130 (+ 863us) server_negotiation.cc:206] Beginning negotiation
0620 14:52:53.667149 (+ 19us) server_negotiation.cc:399] Waiting for connection header
0620 14:52:53.667208 (+ 59us) server_negotiation.cc:407] Connection header received
0620 14:52:53.668995 (+ 1787us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:52:53.669021 (+ 26us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:52:53.669392 (+ 371us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:52:53.669851 (+ 459us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:52:53.670991 (+ 1140us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 14:52:53.670996 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 14:52:53.671010 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 14:52:53.671025 (+ 15us) server_negotiation.cc:922] calling sasl_server_start()
0620 14:52:53.671102 (+ 77us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 14:52:53.671126 (+ 24us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 14:52:53.671389 (+ 263us) server_negotiation.cc:1035] Waiting for connection context
0620 14:52:53.671538 (+ 149us) server_negotiation.cc:299] Negotiation successful
0620 14:52:53.671784 (+ 246us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":590,"thread_start_us":352,"threads_started":1}
I20260620 14:52:53.676867 388 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260620 14:52:53.679764 22086 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 8939d190f17a4cbabb58a59eef2891d2: Bootstrap starting.
I20260620 14:52:53.683996 22089 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:52:53.681449 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.95:57069 (local address 127.0.0.1:40323)
0620 14:52:53.681712 (+ 263us) server_negotiation.cc:206] Beginning negotiation
0620 14:52:53.681731 (+ 19us) server_negotiation.cc:399] Waiting for connection header
0620 14:52:53.681968 (+ 237us) server_negotiation.cc:407] Connection header received
0620 14:52:53.682148 (+ 180us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:52:53.682154 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:52:53.682221 (+ 67us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:52:53.682339 (+ 118us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:52:53.683259 (+ 920us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 14:52:53.683264 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 14:52:53.683269 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 14:52:53.683288 (+ 19us) server_negotiation.cc:922] calling sasl_server_start()
0620 14:52:53.683337 (+ 49us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 14:52:53.683358 (+ 21us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 14:52:53.683477 (+ 119us) server_negotiation.cc:1035] Waiting for connection context
0620 14:52:53.683552 (+ 75us) server_negotiation.cc:299] Negotiation successful
0620 14:52:53.683654 (+ 102us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":94}
I20260620 14:52:53.684073 22086 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 8939d190f17a4cbabb58a59eef2891d2: Neither blocks nor log segments found. Creating new log.
I20260620 14:52:53.685539 22086 log.cc:824] T 00000000000000000000000000000000 P 8939d190f17a4cbabb58a59eef2891d2: Log is configured to *not* fsync() on all Append() calls
I20260620 14:52:53.688596 22086 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 8939d190f17a4cbabb58a59eef2891d2: No bootstrap required, opened a new log
I20260620 14:52:53.696816 22086 raft_consensus.cc:348] T 00000000000000000000000000000000 P 8939d190f17a4cbabb58a59eef2891d2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8939d190f17a4cbabb58a59eef2891d2" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 40323 } }
I20260620 14:52:53.697266 22086 raft_consensus.cc:374] T 00000000000000000000000000000000 P 8939d190f17a4cbabb58a59eef2891d2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 14:52:53.697391 22086 raft_consensus.cc:729] T 00000000000000000000000000000000 P 8939d190f17a4cbabb58a59eef2891d2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8939d190f17a4cbabb58a59eef2891d2, State: Initialized, Role: FOLLOWER
I20260620 14:52:53.698016 22086 consensus_queue.cc:260] T 00000000000000000000000000000000 P 8939d190f17a4cbabb58a59eef2891d2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8939d190f17a4cbabb58a59eef2891d2" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 40323 } }
I20260620 14:52:53.698261 22086 raft_consensus.cc:388] T 00000000000000000000000000000000 P 8939d190f17a4cbabb58a59eef2891d2 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 14:52:53.698421 22086 raft_consensus.cc:482] T 00000000000000000000000000000000 P 8939d190f17a4cbabb58a59eef2891d2 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 14:52:53.698588 22086 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 8939d190f17a4cbabb58a59eef2891d2 [term 0 FOLLOWER]: Advancing to term 1
I20260620 14:52:53.700944 22086 raft_consensus.cc:504] T 00000000000000000000000000000000 P 8939d190f17a4cbabb58a59eef2891d2 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8939d190f17a4cbabb58a59eef2891d2" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 40323 } }
I20260620 14:52:53.701537 22086 leader_election.cc:302] T 00000000000000000000000000000000 P 8939d190f17a4cbabb58a59eef2891d2 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 8939d190f17a4cbabb58a59eef2891d2; no voters:
I20260620 14:52:53.702288 22086 leader_election.cc:288] T 00000000000000000000000000000000 P 8939d190f17a4cbabb58a59eef2891d2 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260620 14:52:53.702404 22096 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 8939d190f17a4cbabb58a59eef2891d2 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 14:52:53.703040 22096 raft_consensus.cc:686] T 00000000000000000000000000000000 P 8939d190f17a4cbabb58a59eef2891d2 [term 1 LEADER]: Becoming Leader. State: Replica: 8939d190f17a4cbabb58a59eef2891d2, State: Running, Role: LEADER
I20260620 14:52:53.703684 22096 consensus_queue.cc:237] T 00000000000000000000000000000000 P 8939d190f17a4cbabb58a59eef2891d2 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8939d190f17a4cbabb58a59eef2891d2" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 40323 } }
I20260620 14:52:53.704787 22086 sys_catalog.cc:564] T 00000000000000000000000000000000 P 8939d190f17a4cbabb58a59eef2891d2 [sys.catalog]: configured and running, proceeding with master startup.
I20260620 14:52:53.706791 22097 sys_catalog.cc:455] T 00000000000000000000000000000000 P 8939d190f17a4cbabb58a59eef2891d2 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "8939d190f17a4cbabb58a59eef2891d2" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8939d190f17a4cbabb58a59eef2891d2" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 40323 } } }
I20260620 14:52:53.707206 22097 sys_catalog.cc:458] T 00000000000000000000000000000000 P 8939d190f17a4cbabb58a59eef2891d2 [sys.catalog]: This master's current role is: LEADER
I20260620 14:52:53.709206 22098 sys_catalog.cc:455] T 00000000000000000000000000000000 P 8939d190f17a4cbabb58a59eef2891d2 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 8939d190f17a4cbabb58a59eef2891d2. Latest consensus state: current_term: 1 leader_uuid: "8939d190f17a4cbabb58a59eef2891d2" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8939d190f17a4cbabb58a59eef2891d2" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 40323 } } }
I20260620 14:52:53.709673 22098 sys_catalog.cc:458] T 00000000000000000000000000000000 P 8939d190f17a4cbabb58a59eef2891d2 [sys.catalog]: This master's current role is: LEADER
I20260620 14:52:53.713284 22104 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 14:52:53.719358 22104 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 14:52:53.729337 22104 catalog_manager.cc:1347] Generated new cluster ID: 6bdaf69dbd8847ee9bab66cfdba9469a
I20260620 14:52:53.729475 22104 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 14:52:53.783445 22104 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 14:52:53.783959 22104 catalog_manager.cc:1504] Loading token signing keys...
I20260620 14:52:53.798254 22104 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 8939d190f17a4cbabb58a59eef2891d2: Generated new TSK 0
I20260620 14:52:53.799221 22104 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260620 14:52:53.806571 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskUci3DU/build/asan/bin/kudu pid 22019
W20260620 14:52:53.807862 22081 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='slave'} at 10.240.0.95:55895
W20260620 14:52:53.808717 388 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskUci3DU/build/asan/bin/kudu pid 22019: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260620 14:52:53.808885 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskUci3DU/build/asan/bin/kudu with pid 22019
2026-06-20T14:52:53Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/11 (1976 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/12
Loading random data
Initializing database '/tmp/dist-test-taskUci3DU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1781966829091300-388-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 20 14:52:53 dist-test-slave-q9j6 krb5kdc[22117](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 20 14:52:53 dist-test-slave-q9j6 krb5kdc[22117](info): set up 2 sockets
Jun 20 14:52:53 dist-test-slave-q9j6 krb5kdc[22117](info): commencing operation
krb5kdc: starting...
W20260620 14:52:54.924271 388 mini_kdc.cc:121] Time spent starting KDC: real 1.078s user 0.006s sys 0.022s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskUci3DU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1781966829091300-388-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskUci3DU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1781966829091300-388-0/krb5kdc/test-user.keytab.
Jun 20 14:52:55 dist-test-slave-q9j6 krb5kdc[22117](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781967175, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-20T14:52:55Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T14:52:55Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.97.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskUci3DU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1781966829091300-388-0/krb5kdc/kudu_127.0.97.62.keytab.
Entry for principal kudu/127.0.97.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskUci3DU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1781966829091300-388-0/krb5kdc/kudu_127.0.97.62.keytab.
I20260620 14:52:55.190858 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskUci3DU/build/asan/bin/kudu
/tmp/dist-test-taskUci3DU/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskUci3DU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1781966829091300-388-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskUci3DU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1781966829091300-388-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskUci3DU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1781966829091300-388-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskUci3DU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1781966829091300-388-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:35389
--webserver_interface=127.0.97.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskUci3DU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1781966829091300-388-0/krb5kdc/kudu_127.0.97.62.keytab
--principal=kudu/127.0.97.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:44867
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.97.62:35389
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskUci3DU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1781966829091300-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskUci3DU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1781966829091300-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskUci3DU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1781966829091300-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 14:52:55.542109 22130 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 14:52:55.542526 22130 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 14:52:55.542714 22130 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 14:52:55.550359 22130 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 14:52:55.550557 22130 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 14:52:55.550675 22130 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 14:52:55.550791 22130 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 14:52:55.550881 22130 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 14:52:55.561662 22130 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:44867
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskUci3DU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1781966829091300-388-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskUci3DU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1781966829091300-388-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.97.62:35389
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskUci3DU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1781966829091300-388-0/krb5kdc/kudu_127.0.97.62.keytab
--principal=kudu/127.0.97.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:35389
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskUci3DU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1781966829091300-388-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.97.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskUci3DU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1781966829091300-388-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 13:41:20 UTC on ccycloud.quasar-vdagoi.root.comops.site
build id 4457462
ASAN enabled
I20260620 14:52:55.563151 22130 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 14:52:55.564610 22130 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 14:52:55.577800 22136 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 14:52:55.577739 22135 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 14:52:55.577780 22138 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 14:52:55.578747 22130 server_base.cc:1034] running on GCE node
I20260620 14:52:55.579712 22130 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 14:52:55.581281 22130 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 14:52:55.582535 22130 hybrid_clock.cc:648] HybridClock initialized: now 1781967175582489 us; error 65 us; skew 500 ppm
I20260620 14:52:55.582950 22130 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 14:52:55 dist-test-slave-q9j6 krb5kdc[22117](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781967175, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 14:52:55.586302 22130 init.cc:377] Logged in from keytab as kudu/127.0.97.62@KRBTEST.COM (short username kudu)
I20260620 14:52:55.588079 22130 webserver.cc:460] Webserver started at http://127.0.97.62:46783/ using document root <none> and password file <none>
I20260620 14:52:55.588794 22130 fs_manager.cc:359] Metadata directory not provided
I20260620 14:52:55.588902 22130 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 14:52:55.589170 22130 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 14:52:55.592509 22130 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskUci3DU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1781966829091300-388-0/minicluster-data/master-0/data/instance:
uuid: "45b50a95c8264a3a9671ba4f781451ae"
format_stamp: "Formatted at 2026-06-20 14:52:55 on dist-test-slave-q9j6"
server_key: "c8f5a29b3444c903d06ecf1770bb8fc3"
server_key_iv: "f9cc2eb2683a4940bc570cdf01ec4097"
server_key_version: "encryptionkey@0"
I20260620 14:52:55.593345 22130 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskUci3DU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1781966829091300-388-0/minicluster-data/master-0/wal/instance:
uuid: "45b50a95c8264a3a9671ba4f781451ae"
format_stamp: "Formatted at 2026-06-20 14:52:55 on dist-test-slave-q9j6"
server_key: "c8f5a29b3444c903d06ecf1770bb8fc3"
server_key_iv: "f9cc2eb2683a4940bc570cdf01ec4097"
server_key_version: "encryptionkey@0"
I20260620 14:52:55.599146 22130 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.000s
I20260620 14:52:55.603106 22145 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 14:52:55.604736 22130 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260620 14:52:55.604903 22130 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskUci3DU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1781966829091300-388-0/minicluster-data/master-0/data,/tmp/dist-test-taskUci3DU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1781966829091300-388-0/minicluster-data/master-0/wal
uuid: "45b50a95c8264a3a9671ba4f781451ae"
format_stamp: "Formatted at 2026-06-20 14:52:55 on dist-test-slave-q9j6"
server_key: "c8f5a29b3444c903d06ecf1770bb8fc3"
server_key_iv: "f9cc2eb2683a4940bc570cdf01ec4097"
server_key_version: "encryptionkey@0"
I20260620 14:52:55.605091 22130 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskUci3DU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1781966829091300-388-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskUci3DU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1781966829091300-388-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskUci3DU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1781966829091300-388-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 14:52:55.622906 22130 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 14:52:55.627447 22130 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 14:52:55.627831 22130 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 14:52:55.647229 22130 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.62:35389
I20260620 14:52:55.647305 22196 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.62:35389 every 8 connection(s)
I20260620 14:52:55.648823 22130 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskUci3DU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1781966829091300-388-0/minicluster-data/master-0/data/info.pb
I20260620 14:52:55.653111 22197 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 14:52:55.657583 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskUci3DU/build/asan/bin/kudu as pid 22130
I20260620 14:52:55.657781 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskUci3DU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1781966829091300-388-0/minicluster-data/master-0/wal/instance
I20260620 14:52:55.658210 388 external_mini_cluster.cc:1428] Setting key e2df88b11e6ee329fa44e53d5a91a5e9
I20260620 14:52:55.666370 22197 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 45b50a95c8264a3a9671ba4f781451ae: Bootstrap starting.
I20260620 14:52:55.670378 22197 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 45b50a95c8264a3a9671ba4f781451ae: Neither blocks nor log segments found. Creating new log.
I20260620 14:52:55.671567 22197 log.cc:824] T 00000000000000000000000000000000 P 45b50a95c8264a3a9671ba4f781451ae: Log is configured to *not* fsync() on all Append() calls
I20260620 14:52:55.674649 22197 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 45b50a95c8264a3a9671ba4f781451ae: No bootstrap required, opened a new log
Jun 20 14:52:55 dist-test-slave-q9j6 krb5kdc[22117](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781967175, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260620 14:52:55.685832 22200 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:52:55.660267 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49631 (local address 127.0.97.62:35389)
0620 14:52:55.661179 (+ 912us) server_negotiation.cc:206] Beginning negotiation
0620 14:52:55.661196 (+ 17us) server_negotiation.cc:399] Waiting for connection header
0620 14:52:55.661265 (+ 69us) server_negotiation.cc:407] Connection header received
0620 14:52:55.664417 (+ 3152us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:52:55.664455 (+ 38us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:52:55.665084 (+ 629us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:52:55.665617 (+ 533us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:52:55.667066 (+ 1449us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:52:55.669183 (+ 2117us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:52:55.670605 (+ 1422us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:52:55.671387 (+ 782us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:52:55.671676 (+ 289us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:52:55.676015 (+ 4339us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 14:52:55.676027 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 14:52:55.676048 (+ 21us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 14:52:55.676103 (+ 55us) server_negotiation.cc:922] calling sasl_server_start()
0620 14:52:55.680329 (+ 4226us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:52:55.681336 (+ 1007us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:52:55.681345 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:52:55.681355 (+ 10us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:52:55.681604 (+ 249us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:52:55.682081 (+ 477us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:52:55.682088 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:52:55.682096 (+ 8us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:52:55.682946 (+ 850us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 14:52:55.683314 (+ 368us) server_negotiation.cc:1035] Waiting for connection context
0620 14:52:55.683832 (+ 518us) server_negotiation.cc:299] Negotiation successful
0620 14:52:55.684264 (+ 432us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":636,"thread_start_us":397,"threads_started":1}
I20260620 14:52:55.686175 22197 raft_consensus.cc:348] T 00000000000000000000000000000000 P 45b50a95c8264a3a9671ba4f781451ae [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "45b50a95c8264a3a9671ba4f781451ae" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 35389 } }
I20260620 14:52:55.686689 22197 raft_consensus.cc:374] T 00000000000000000000000000000000 P 45b50a95c8264a3a9671ba4f781451ae [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 14:52:55.686893 22197 raft_consensus.cc:729] T 00000000000000000000000000000000 P 45b50a95c8264a3a9671ba4f781451ae [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 45b50a95c8264a3a9671ba4f781451ae, State: Initialized, Role: FOLLOWER
I20260620 14:52:55.688169 22197 consensus_queue.cc:260] T 00000000000000000000000000000000 P 45b50a95c8264a3a9671ba4f781451ae [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "45b50a95c8264a3a9671ba4f781451ae" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 35389 } }
I20260620 14:52:55.688511 22197 raft_consensus.cc:388] T 00000000000000000000000000000000 P 45b50a95c8264a3a9671ba4f781451ae [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 14:52:55.688665 22197 raft_consensus.cc:482] T 00000000000000000000000000000000 P 45b50a95c8264a3a9671ba4f781451ae [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 14:52:55.688871 22197 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 45b50a95c8264a3a9671ba4f781451ae [term 0 FOLLOWER]: Advancing to term 1
I20260620 14:52:55.691362 388 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260620 14:52:55.692716 22197 raft_consensus.cc:504] T 00000000000000000000000000000000 P 45b50a95c8264a3a9671ba4f781451ae [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "45b50a95c8264a3a9671ba4f781451ae" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 35389 } }
I20260620 14:52:55.693462 22197 leader_election.cc:302] T 00000000000000000000000000000000 P 45b50a95c8264a3a9671ba4f781451ae [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 45b50a95c8264a3a9671ba4f781451ae; no voters:
I20260620 14:52:55.694244 22197 leader_election.cc:288] T 00000000000000000000000000000000 P 45b50a95c8264a3a9671ba4f781451ae [CANDIDATE]: Term 1 election: Requested vote from peers
I20260620 14:52:55.694384 22202 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 45b50a95c8264a3a9671ba4f781451ae [term 1 FOLLOWER]: Leader election won for term 1
I20260620 14:52:55.697511 22197 sys_catalog.cc:564] T 00000000000000000000000000000000 P 45b50a95c8264a3a9671ba4f781451ae [sys.catalog]: configured and running, proceeding with master startup.
I20260620 14:52:55.699275 22202 raft_consensus.cc:686] T 00000000000000000000000000000000 P 45b50a95c8264a3a9671ba4f781451ae [term 1 LEADER]: Becoming Leader. State: Replica: 45b50a95c8264a3a9671ba4f781451ae, State: Running, Role: LEADER
I20260620 14:52:55.700161 22202 consensus_queue.cc:237] T 00000000000000000000000000000000 P 45b50a95c8264a3a9671ba4f781451ae [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "45b50a95c8264a3a9671ba4f781451ae" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 35389 } }
I20260620 14:52:55.704315 22203 sys_catalog.cc:455] T 00000000000000000000000000000000 P 45b50a95c8264a3a9671ba4f781451ae [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "45b50a95c8264a3a9671ba4f781451ae" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "45b50a95c8264a3a9671ba4f781451ae" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 35389 } } }
I20260620 14:52:55.704856 22203 sys_catalog.cc:458] T 00000000000000000000000000000000 P 45b50a95c8264a3a9671ba4f781451ae [sys.catalog]: This master's current role is: LEADER
I20260620 14:52:55.708127 22204 sys_catalog.cc:455] T 00000000000000000000000000000000 P 45b50a95c8264a3a9671ba4f781451ae [sys.catalog]: SysCatalogTable state changed. Reason: New leader 45b50a95c8264a3a9671ba4f781451ae. Latest consensus state: current_term: 1 leader_uuid: "45b50a95c8264a3a9671ba4f781451ae" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "45b50a95c8264a3a9671ba4f781451ae" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 35389 } } }
I20260620 14:52:55.708693 22204 sys_catalog.cc:458] T 00000000000000000000000000000000 P 45b50a95c8264a3a9671ba4f781451ae [sys.catalog]: This master's current role is: LEADER
I20260620 14:52:55.710778 22210 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 14:52:55.720881 22210 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 14:52:55.726960 22224 catalog_manager.cc:795] Waiting for catalog manager background task thread to start: Service unavailable: Catalog manager is not initialized. State: Starting
I20260620 14:52:55.732131 22200 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:52:55.711222 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57921 (local address 127.0.97.62:35389)
0620 14:52:55.711436 (+ 214us) server_negotiation.cc:206] Beginning negotiation
0620 14:52:55.711445 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0620 14:52:55.712196 (+ 751us) server_negotiation.cc:407] Connection header received
0620 14:52:55.712425 (+ 229us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:52:55.712430 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:52:55.712517 (+ 87us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:52:55.712656 (+ 139us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:52:55.714604 (+ 1948us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:52:55.715421 (+ 817us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:52:55.717501 (+ 2080us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:52:55.718217 (+ 716us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:52:55.718335 (+ 118us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:52:55.722112 (+ 3777us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 14:52:55.722121 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 14:52:55.722127 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 14:52:55.722180 (+ 53us) server_negotiation.cc:922] calling sasl_server_start()
0620 14:52:55.726802 (+ 4622us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:52:55.728007 (+ 1205us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:52:55.728013 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:52:55.728017 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:52:55.728180 (+ 163us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:52:55.728804 (+ 624us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:52:55.728812 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:52:55.728818 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:52:55.729287 (+ 469us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 14:52:55.731197 (+ 1910us) server_negotiation.cc:1035] Waiting for connection context
0620 14:52:55.731415 (+ 218us) server_negotiation.cc:299] Negotiation successful
0620 14:52:55.731720 (+ 305us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":66}
I20260620 14:52:55.733623 22210 catalog_manager.cc:1347] Generated new cluster ID: 811d1ffbf49147b8b2951b2e803561eb
I20260620 14:52:55.733736 22210 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 14:52:55.767376 22210 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 14:52:55.768882 22210 catalog_manager.cc:1504] Loading token signing keys...
I20260620 14:52:55.782572 22210 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 45b50a95c8264a3a9671ba4f781451ae: Generated new TSK 0
I20260620 14:52:55.783942 22210 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260620 14:52:55.800127 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskUci3DU/build/asan/bin/kudu pid 22130
I20260620 14:52:55.801522 22192 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 14:52:55.897573 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskUci3DU/build/asan/bin/kudu with pid 22130
W20260620 14:52:55.915575 22115 connection.cc:535] client connection to 127.0.97.62:35389 recv error: Network error: failed to read from TLS socket (remote: 127.0.97.62:35389): Connection reset by peer (error 104)
2026-06-20T14:52:55Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/12 (2089 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/13
Loading random data
Initializing database '/tmp/dist-test-taskUci3DU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1781966829091300-388-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 20 14:52:55 dist-test-slave-q9j6 krb5kdc[22229](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 20 14:52:55 dist-test-slave-q9j6 krb5kdc[22229](info): set up 2 sockets
Jun 20 14:52:55 dist-test-slave-q9j6 krb5kdc[22229](info): commencing operation
krb5kdc: starting...
W20260620 14:52:57.020174 388 mini_kdc.cc:121] Time spent starting KDC: real 1.085s user 0.007s sys 0.022s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskUci3DU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1781966829091300-388-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskUci3DU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1781966829091300-388-0/krb5kdc/test-user.keytab.
Jun 20 14:52:57 dist-test-slave-q9j6 krb5kdc[22229](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781967177, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-20T14:52:57Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T14:52:57Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.97.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskUci3DU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1781966829091300-388-0/krb5kdc/kudu_127.0.97.62.keytab.
Entry for principal kudu/127.0.97.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskUci3DU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1781966829091300-388-0/krb5kdc/kudu_127.0.97.62.keytab.
I20260620 14:52:57.291286 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskUci3DU/build/asan/bin/kudu
/tmp/dist-test-taskUci3DU/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskUci3DU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1781966829091300-388-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskUci3DU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1781966829091300-388-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskUci3DU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1781966829091300-388-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskUci3DU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1781966829091300-388-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:44037
--webserver_interface=127.0.97.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskUci3DU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1781966829091300-388-0/krb5kdc/kudu_127.0.97.62.keytab
--principal=kudu/127.0.97.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:43983
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.97.62:44037
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskUci3DU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1781966829091300-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskUci3DU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1781966829091300-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskUci3DU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1781966829091300-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 14:52:57.652829 22242 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 14:52:57.653180 22242 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 14:52:57.653522 22242 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 14:52:57.661535 22242 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 14:52:57.661695 22242 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 14:52:57.661803 22242 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 14:52:57.661929 22242 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 14:52:57.662012 22242 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 14:52:57.672811 22242 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:43983
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskUci3DU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1781966829091300-388-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskUci3DU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1781966829091300-388-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.97.62:44037
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskUci3DU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1781966829091300-388-0/krb5kdc/kudu_127.0.97.62.keytab
--principal=kudu/127.0.97.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:44037
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskUci3DU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1781966829091300-388-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.97.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskUci3DU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1781966829091300-388-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 13:41:20 UTC on ccycloud.quasar-vdagoi.root.comops.site
build id 4457462
ASAN enabled
I20260620 14:52:57.674266 22242 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 14:52:57.675689 22242 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 14:52:57.688431 22247 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 14:52:57.688357 22250 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 14:52:57.688382 22248 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 14:52:57.689626 22242 server_base.cc:1034] running on GCE node
I20260620 14:52:57.690548 22242 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 14:52:57.692179 22242 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 14:52:57.693396 22242 hybrid_clock.cc:648] HybridClock initialized: now 1781967177693330 us; error 58 us; skew 500 ppm
I20260620 14:52:57.693794 22242 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 14:52:57 dist-test-slave-q9j6 krb5kdc[22229](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781967177, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 14:52:57.697100 22242 init.cc:377] Logged in from keytab as kudu/127.0.97.62@KRBTEST.COM (short username kudu)
I20260620 14:52:57.698736 22242 webserver.cc:460] Webserver started at http://127.0.97.62:44611/ using document root <none> and password file <none>
I20260620 14:52:57.699504 22242 fs_manager.cc:359] Metadata directory not provided
I20260620 14:52:57.699680 22242 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 14:52:57.700026 22242 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 14:52:57.703311 22242 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskUci3DU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1781966829091300-388-0/minicluster-data/master-0/data/instance:
uuid: "860176760c49439e94fe2cc3feef1291"
format_stamp: "Formatted at 2026-06-20 14:52:57 on dist-test-slave-q9j6"
server_key: "dbcd5a717501f674a965010f090e194c"
server_key_iv: "fa12e035fc3cd6592821f0cf4a1582b1"
server_key_version: "encryptionkey@0"
I20260620 14:52:57.704281 22242 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskUci3DU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1781966829091300-388-0/minicluster-data/master-0/wal/instance:
uuid: "860176760c49439e94fe2cc3feef1291"
format_stamp: "Formatted at 2026-06-20 14:52:57 on dist-test-slave-q9j6"
server_key: "dbcd5a717501f674a965010f090e194c"
server_key_iv: "fa12e035fc3cd6592821f0cf4a1582b1"
server_key_version: "encryptionkey@0"
I20260620 14:52:57.710210 22242 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.001s sys 0.004s
I20260620 14:52:57.714107 22257 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 14:52:57.715694 22242 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260620 14:52:57.715919 22242 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskUci3DU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1781966829091300-388-0/minicluster-data/master-0/data,/tmp/dist-test-taskUci3DU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1781966829091300-388-0/minicluster-data/master-0/wal
uuid: "860176760c49439e94fe2cc3feef1291"
format_stamp: "Formatted at 2026-06-20 14:52:57 on dist-test-slave-q9j6"
server_key: "dbcd5a717501f674a965010f090e194c"
server_key_iv: "fa12e035fc3cd6592821f0cf4a1582b1"
server_key_version: "encryptionkey@0"
I20260620 14:52:57.716166 22242 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskUci3DU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1781966829091300-388-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskUci3DU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1781966829091300-388-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskUci3DU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1781966829091300-388-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 14:52:57.738881 22242 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 14:52:57.743242 22242 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 14:52:57.743557 22242 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 14:52:57.763855 22242 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.62:44037
I20260620 14:52:57.763880 22308 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.62:44037 every 8 connection(s)
I20260620 14:52:57.765336 22242 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskUci3DU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1781966829091300-388-0/minicluster-data/master-0/data/info.pb
I20260620 14:52:57.768826 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskUci3DU/build/asan/bin/kudu as pid 22242
I20260620 14:52:57.769023 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskUci3DU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1781966829091300-388-0/minicluster-data/master-0/wal/instance
I20260620 14:52:57.769513 388 external_mini_cluster.cc:1428] Setting key f1e7705b5f2bdc5e834f2b2523243366
I20260620 14:52:57.769713 22309 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 14:52:57.785862 22309 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 860176760c49439e94fe2cc3feef1291: Bootstrap starting.
I20260620 14:52:57.786077 22312 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:52:57.771449 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51387 (local address 127.0.97.62:44037)
0620 14:52:57.772361 (+ 912us) server_negotiation.cc:206] Beginning negotiation
0620 14:52:57.772377 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0620 14:52:57.772434 (+ 57us) server_negotiation.cc:407] Connection header received
0620 14:52:57.774833 (+ 2399us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:52:57.774867 (+ 34us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:52:57.775500 (+ 633us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:52:57.776033 (+ 533us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:52:57.776626 (+ 593us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:52:57.778729 (+ 2103us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:52:57.780342 (+ 1613us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:52:57.781482 (+ 1140us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:52:57.781695 (+ 213us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:52:57.783046 (+ 1351us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 14:52:57.783057 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 14:52:57.783078 (+ 21us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 14:52:57.783905 (+ 827us) server_negotiation.cc:922] calling sasl_server_start()
0620 14:52:57.784006 (+ 101us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 14:52:57.784041 (+ 35us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 14:52:57.784226 (+ 185us) server_negotiation.cc:1035] Waiting for connection context
0620 14:52:57.784480 (+ 254us) server_negotiation.cc:299] Negotiation successful
0620 14:52:57.784680 (+ 200us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":653,"thread_start_us":413,"threads_started":1}
I20260620 14:52:57.789579 22309 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 860176760c49439e94fe2cc3feef1291: Neither blocks nor log segments found. Creating new log.
I20260620 14:52:57.790423 388 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260620 14:52:57.790832 22309 log.cc:824] T 00000000000000000000000000000000 P 860176760c49439e94fe2cc3feef1291: Log is configured to *not* fsync() on all Append() calls
I20260620 14:52:57.794133 22309 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 860176760c49439e94fe2cc3feef1291: No bootstrap required, opened a new log
I20260620 14:52:57.802704 22312 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:52:57.795454 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33881 (local address 127.0.97.62:44037)
0620 14:52:57.795668 (+ 214us) server_negotiation.cc:206] Beginning negotiation
0620 14:52:57.795674 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 14:52:57.796071 (+ 397us) server_negotiation.cc:407] Connection header received
0620 14:52:57.796227 (+ 156us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:52:57.796232 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:52:57.796315 (+ 83us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:52:57.796444 (+ 129us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:52:57.796959 (+ 515us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:52:57.798173 (+ 1214us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:52:57.799776 (+ 1603us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:52:57.800509 (+ 733us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:52:57.800665 (+ 156us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:52:57.801830 (+ 1165us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 14:52:57.801837 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 14:52:57.801842 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 14:52:57.801868 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0620 14:52:57.801942 (+ 74us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 14:52:57.801963 (+ 21us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 14:52:57.802103 (+ 140us) server_negotiation.cc:1035] Waiting for connection context
0620 14:52:57.802246 (+ 143us) server_negotiation.cc:299] Negotiation successful
0620 14:52:57.802317 (+ 71us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":64}
I20260620 14:52:57.804168 22309 raft_consensus.cc:348] T 00000000000000000000000000000000 P 860176760c49439e94fe2cc3feef1291 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "860176760c49439e94fe2cc3feef1291" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 44037 } }
I20260620 14:52:57.804718 22309 raft_consensus.cc:374] T 00000000000000000000000000000000 P 860176760c49439e94fe2cc3feef1291 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 14:52:57.804826 22309 raft_consensus.cc:729] T 00000000000000000000000000000000 P 860176760c49439e94fe2cc3feef1291 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 860176760c49439e94fe2cc3feef1291, State: Initialized, Role: FOLLOWER
I20260620 14:52:57.805397 22309 consensus_queue.cc:260] T 00000000000000000000000000000000 P 860176760c49439e94fe2cc3feef1291 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "860176760c49439e94fe2cc3feef1291" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 44037 } }
I20260620 14:52:57.805625 22309 raft_consensus.cc:388] T 00000000000000000000000000000000 P 860176760c49439e94fe2cc3feef1291 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 14:52:57.805732 22309 raft_consensus.cc:482] T 00000000000000000000000000000000 P 860176760c49439e94fe2cc3feef1291 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 14:52:57.805868 22309 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 860176760c49439e94fe2cc3feef1291 [term 0 FOLLOWER]: Advancing to term 1
I20260620 14:52:57.808550 22309 raft_consensus.cc:504] T 00000000000000000000000000000000 P 860176760c49439e94fe2cc3feef1291 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "860176760c49439e94fe2cc3feef1291" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 44037 } }
I20260620 14:52:57.809226 22309 leader_election.cc:302] T 00000000000000000000000000000000 P 860176760c49439e94fe2cc3feef1291 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 860176760c49439e94fe2cc3feef1291; no voters:
I20260620 14:52:57.809901 22309 leader_election.cc:288] T 00000000000000000000000000000000 P 860176760c49439e94fe2cc3feef1291 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260620 14:52:57.810058 22319 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 860176760c49439e94fe2cc3feef1291 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 14:52:57.810802 22319 raft_consensus.cc:686] T 00000000000000000000000000000000 P 860176760c49439e94fe2cc3feef1291 [term 1 LEADER]: Becoming Leader. State: Replica: 860176760c49439e94fe2cc3feef1291, State: Running, Role: LEADER
I20260620 14:52:57.811381 22319 consensus_queue.cc:237] T 00000000000000000000000000000000 P 860176760c49439e94fe2cc3feef1291 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "860176760c49439e94fe2cc3feef1291" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 44037 } }
I20260620 14:52:57.812206 22309 sys_catalog.cc:564] T 00000000000000000000000000000000 P 860176760c49439e94fe2cc3feef1291 [sys.catalog]: configured and running, proceeding with master startup.
I20260620 14:52:57.814545 22320 sys_catalog.cc:455] T 00000000000000000000000000000000 P 860176760c49439e94fe2cc3feef1291 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "860176760c49439e94fe2cc3feef1291" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "860176760c49439e94fe2cc3feef1291" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 44037 } } }
I20260620 14:52:57.815035 22320 sys_catalog.cc:458] T 00000000000000000000000000000000 P 860176760c49439e94fe2cc3feef1291 [sys.catalog]: This master's current role is: LEADER
I20260620 14:52:57.814523 22321 sys_catalog.cc:455] T 00000000000000000000000000000000 P 860176760c49439e94fe2cc3feef1291 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 860176760c49439e94fe2cc3feef1291. Latest consensus state: current_term: 1 leader_uuid: "860176760c49439e94fe2cc3feef1291" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "860176760c49439e94fe2cc3feef1291" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 44037 } } }
I20260620 14:52:57.815492 22321 sys_catalog.cc:458] T 00000000000000000000000000000000 P 860176760c49439e94fe2cc3feef1291 [sys.catalog]: This master's current role is: LEADER
I20260620 14:52:57.815888 22324 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 14:52:57.822006 22324 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 14:52:57.832553 22324 catalog_manager.cc:1347] Generated new cluster ID: f09d9d49d85e4a2c8d5f217b50dd73f8
I20260620 14:52:57.832708 22324 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 14:52:57.862612 22324 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 14:52:57.864185 22324 catalog_manager.cc:1504] Loading token signing keys...
I20260620 14:52:57.879349 22324 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 860176760c49439e94fe2cc3feef1291: Generated new TSK 0
I20260620 14:52:57.880627 22324 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260620 14:52:57.890780 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskUci3DU/build/asan/bin/kudu pid 22242
W20260620 14:52:57.892072 22304 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='slave'} at 127.0.0.1:51387
W20260620 14:52:57.892925 388 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskUci3DU/build/asan/bin/kudu pid 22242: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260620 14:52:57.893096 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskUci3DU/build/asan/bin/kudu with pid 22242
W20260620 14:52:57.910688 22227 connection.cc:535] client connection to 127.0.97.62:44037 recv error: Network error: failed to read from TLS socket (remote: 127.0.97.62:44037): Connection reset by peer (error 104)
2026-06-20T14:52:57Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/13 (1994 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/14
Loading random data
Initializing database '/tmp/dist-test-taskUci3DU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1781966829091300-388-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 20 14:52:57 dist-test-slave-q9j6 krb5kdc[22340](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 20 14:52:57 dist-test-slave-q9j6 krb5kdc[22340](info): set up 2 sockets
Jun 20 14:52:57 dist-test-slave-q9j6 krb5kdc[22340](info): commencing operation
krb5kdc: starting...
W20260620 14:52:59.012235 388 mini_kdc.cc:121] Time spent starting KDC: real 1.082s user 0.002s sys 0.026s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskUci3DU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1781966829091300-388-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskUci3DU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1781966829091300-388-0/krb5kdc/test-user.keytab.
Jun 20 14:52:59 dist-test-slave-q9j6 krb5kdc[22340](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781967179, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-20T14:52:59Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T14:52:59Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.97.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskUci3DU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1781966829091300-388-0/krb5kdc/kudu_127.0.97.62.keytab.
Entry for principal kudu/127.0.97.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskUci3DU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1781966829091300-388-0/krb5kdc/kudu_127.0.97.62.keytab.
I20260620 14:52:59.283692 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskUci3DU/build/asan/bin/kudu
/tmp/dist-test-taskUci3DU/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskUci3DU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1781966829091300-388-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskUci3DU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1781966829091300-388-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskUci3DU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1781966829091300-388-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskUci3DU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1781966829091300-388-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:42589
--webserver_interface=127.0.97.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskUci3DU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1781966829091300-388-0/krb5kdc/kudu_127.0.97.62.keytab
--principal=kudu/127.0.97.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:37853
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.97.62:42589
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskUci3DU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1781966829091300-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskUci3DU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1781966829091300-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskUci3DU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1781966829091300-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 14:52:59.630357 22353 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 14:52:59.630686 22353 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 14:52:59.630877 22353 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 14:52:59.638353 22353 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 14:52:59.638480 22353 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 14:52:59.638610 22353 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 14:52:59.638723 22353 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 14:52:59.638789 22353 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 14:52:59.649039 22353 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:37853
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskUci3DU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1781966829091300-388-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskUci3DU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1781966829091300-388-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.97.62:42589
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskUci3DU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1781966829091300-388-0/krb5kdc/kudu_127.0.97.62.keytab
--principal=kudu/127.0.97.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:42589
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskUci3DU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1781966829091300-388-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.97.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskUci3DU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1781966829091300-388-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 13:41:20 UTC on ccycloud.quasar-vdagoi.root.comops.site
build id 4457462
ASAN enabled
I20260620 14:52:59.650435 22353 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 14:52:59.651880 22353 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 14:52:59.664490 22358 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 14:52:59.664389 22359 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 14:52:59.664574 22361 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 14:52:59.665783 22353 server_base.cc:1034] running on GCE node
I20260620 14:52:59.666795 22353 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 14:52:59.668447 22353 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 14:52:59.669706 22353 hybrid_clock.cc:648] HybridClock initialized: now 1781967179669635 us; error 71 us; skew 500 ppm
I20260620 14:52:59.670092 22353 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 14:52:59 dist-test-slave-q9j6 krb5kdc[22340](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781967179, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 14:52:59.673482 22353 init.cc:377] Logged in from keytab as kudu/127.0.97.62@KRBTEST.COM (short username kudu)
I20260620 14:52:59.675189 22353 webserver.cc:460] Webserver started at http://127.0.97.62:36133/ using document root <none> and password file <none>
I20260620 14:52:59.676014 22353 fs_manager.cc:359] Metadata directory not provided
I20260620 14:52:59.676168 22353 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 14:52:59.676496 22353 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 14:52:59.679822 22353 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskUci3DU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1781966829091300-388-0/minicluster-data/master-0/data/instance:
uuid: "6bf2c7936e3c4602b461d7a128022b8b"
format_stamp: "Formatted at 2026-06-20 14:52:59 on dist-test-slave-q9j6"
server_key: "37c2fa7180ca488df8a51745f58ef869"
server_key_iv: "cc3d8c9a4519715ff5f39a725f190caa"
server_key_version: "encryptionkey@0"
I20260620 14:52:59.680747 22353 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskUci3DU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1781966829091300-388-0/minicluster-data/master-0/wal/instance:
uuid: "6bf2c7936e3c4602b461d7a128022b8b"
format_stamp: "Formatted at 2026-06-20 14:52:59 on dist-test-slave-q9j6"
server_key: "37c2fa7180ca488df8a51745f58ef869"
server_key_iv: "cc3d8c9a4519715ff5f39a725f190caa"
server_key_version: "encryptionkey@0"
I20260620 14:52:59.686496 22353 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.001s
I20260620 14:52:59.690318 22368 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 14:52:59.691823 22353 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.002s
I20260620 14:52:59.692026 22353 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskUci3DU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1781966829091300-388-0/minicluster-data/master-0/data,/tmp/dist-test-taskUci3DU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1781966829091300-388-0/minicluster-data/master-0/wal
uuid: "6bf2c7936e3c4602b461d7a128022b8b"
format_stamp: "Formatted at 2026-06-20 14:52:59 on dist-test-slave-q9j6"
server_key: "37c2fa7180ca488df8a51745f58ef869"
server_key_iv: "cc3d8c9a4519715ff5f39a725f190caa"
server_key_version: "encryptionkey@0"
I20260620 14:52:59.692257 22353 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskUci3DU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1781966829091300-388-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskUci3DU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1781966829091300-388-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskUci3DU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1781966829091300-388-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 14:52:59.698416 22353 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 14:52:59.702816 22353 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 14:52:59.703156 22353 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 14:52:59.723057 22353 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.62:42589
I20260620 14:52:59.723090 22419 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.62:42589 every 8 connection(s)
I20260620 14:52:59.724658 22353 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskUci3DU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1781966829091300-388-0/minicluster-data/master-0/data/info.pb
I20260620 14:52:59.728693 22420 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 14:52:59.730382 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskUci3DU/build/asan/bin/kudu as pid 22353
I20260620 14:52:59.730582 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskUci3DU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1781966829091300-388-0/minicluster-data/master-0/wal/instance
I20260620 14:52:59.731117 388 external_mini_cluster.cc:1428] Setting key 1de8d05baae062a7d28f3d6fdfa4d243
I20260620 14:52:59.739187 22423 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:52:59.733031 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38991 (local address 127.0.97.62:42589)
0620 14:52:59.733888 (+ 857us) server_negotiation.cc:206] Beginning negotiation
0620 14:52:59.733901 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0620 14:52:59.733950 (+ 49us) server_negotiation.cc:407] Connection header received
0620 14:52:59.735646 (+ 1696us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:52:59.735683 (+ 37us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:52:59.736087 (+ 404us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:52:59.736460 (+ 373us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:52:59.737581 (+ 1121us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 14:52:59.737587 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 14:52:59.737601 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 14:52:59.737616 (+ 15us) server_negotiation.cc:922] calling sasl_server_start()
0620 14:52:59.737676 (+ 60us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 14:52:59.737700 (+ 24us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 14:52:59.737872 (+ 172us) server_negotiation.cc:1035] Waiting for connection context
0620 14:52:59.737999 (+ 127us) server_negotiation.cc:299] Negotiation successful
0620 14:52:59.738214 (+ 215us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":608,"thread_start_us":376,"threads_started":1}
I20260620 14:52:59.742920 22420 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 6bf2c7936e3c4602b461d7a128022b8b: Bootstrap starting.
I20260620 14:52:59.743150 388 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260620 14:52:59.747088 22420 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 6bf2c7936e3c4602b461d7a128022b8b: Neither blocks nor log segments found. Creating new log.
I20260620 14:52:59.748584 22420 log.cc:824] T 00000000000000000000000000000000 P 6bf2c7936e3c4602b461d7a128022b8b: Log is configured to *not* fsync() on all Append() calls
I20260620 14:52:59.751456 22423 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:52:59.748667 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36923 (local address 127.0.97.62:42589)
0620 14:52:59.748873 (+ 206us) server_negotiation.cc:206] Beginning negotiation
0620 14:52:59.748882 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0620 14:52:59.749464 (+ 582us) server_negotiation.cc:407] Connection header received
0620 14:52:59.749698 (+ 234us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:52:59.749702 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:52:59.749755 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:52:59.749860 (+ 105us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:52:59.750828 (+ 968us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 14:52:59.750833 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 14:52:59.750836 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 14:52:59.750855 (+ 19us) server_negotiation.cc:922] calling sasl_server_start()
0620 14:52:59.750892 (+ 37us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 14:52:59.750908 (+ 16us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 14:52:59.751032 (+ 124us) server_negotiation.cc:1035] Waiting for connection context
0620 14:52:59.751133 (+ 101us) server_negotiation.cc:299] Negotiation successful
0620 14:52:59.751196 (+ 63us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":42}
I20260620 14:52:59.751979 22420 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 6bf2c7936e3c4602b461d7a128022b8b: No bootstrap required, opened a new log
I20260620 14:52:59.760048 22420 raft_consensus.cc:348] T 00000000000000000000000000000000 P 6bf2c7936e3c4602b461d7a128022b8b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6bf2c7936e3c4602b461d7a128022b8b" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 42589 } }
I20260620 14:52:59.760496 22420 raft_consensus.cc:374] T 00000000000000000000000000000000 P 6bf2c7936e3c4602b461d7a128022b8b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 14:52:59.760625 22420 raft_consensus.cc:729] T 00000000000000000000000000000000 P 6bf2c7936e3c4602b461d7a128022b8b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6bf2c7936e3c4602b461d7a128022b8b, State: Initialized, Role: FOLLOWER
I20260620 14:52:59.761258 22420 consensus_queue.cc:260] T 00000000000000000000000000000000 P 6bf2c7936e3c4602b461d7a128022b8b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6bf2c7936e3c4602b461d7a128022b8b" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 42589 } }
I20260620 14:52:59.761516 22420 raft_consensus.cc:388] T 00000000000000000000000000000000 P 6bf2c7936e3c4602b461d7a128022b8b [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 14:52:59.761674 22420 raft_consensus.cc:482] T 00000000000000000000000000000000 P 6bf2c7936e3c4602b461d7a128022b8b [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 14:52:59.761859 22420 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 6bf2c7936e3c4602b461d7a128022b8b [term 0 FOLLOWER]: Advancing to term 1
I20260620 14:52:59.764622 22420 raft_consensus.cc:504] T 00000000000000000000000000000000 P 6bf2c7936e3c4602b461d7a128022b8b [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6bf2c7936e3c4602b461d7a128022b8b" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 42589 } }
I20260620 14:52:59.765179 22420 leader_election.cc:302] T 00000000000000000000000000000000 P 6bf2c7936e3c4602b461d7a128022b8b [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 6bf2c7936e3c4602b461d7a128022b8b; no voters:
I20260620 14:52:59.765796 22420 leader_election.cc:288] T 00000000000000000000000000000000 P 6bf2c7936e3c4602b461d7a128022b8b [CANDIDATE]: Term 1 election: Requested vote from peers
I20260620 14:52:59.765957 22430 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 6bf2c7936e3c4602b461d7a128022b8b [term 1 FOLLOWER]: Leader election won for term 1
I20260620 14:52:59.766642 22430 raft_consensus.cc:686] T 00000000000000000000000000000000 P 6bf2c7936e3c4602b461d7a128022b8b [term 1 LEADER]: Becoming Leader. State: Replica: 6bf2c7936e3c4602b461d7a128022b8b, State: Running, Role: LEADER
I20260620 14:52:59.767191 22430 consensus_queue.cc:237] T 00000000000000000000000000000000 P 6bf2c7936e3c4602b461d7a128022b8b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6bf2c7936e3c4602b461d7a128022b8b" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 42589 } }
I20260620 14:52:59.768033 22420 sys_catalog.cc:564] T 00000000000000000000000000000000 P 6bf2c7936e3c4602b461d7a128022b8b [sys.catalog]: configured and running, proceeding with master startup.
I20260620 14:52:59.770311 22432 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6bf2c7936e3c4602b461d7a128022b8b [sys.catalog]: SysCatalogTable state changed. Reason: New leader 6bf2c7936e3c4602b461d7a128022b8b. Latest consensus state: current_term: 1 leader_uuid: "6bf2c7936e3c4602b461d7a128022b8b" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6bf2c7936e3c4602b461d7a128022b8b" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 42589 } } }
I20260620 14:52:59.770279 22431 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6bf2c7936e3c4602b461d7a128022b8b [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "6bf2c7936e3c4602b461d7a128022b8b" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6bf2c7936e3c4602b461d7a128022b8b" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 42589 } } }
I20260620 14:52:59.770800 22432 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6bf2c7936e3c4602b461d7a128022b8b [sys.catalog]: This master's current role is: LEADER
I20260620 14:52:59.770814 22431 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6bf2c7936e3c4602b461d7a128022b8b [sys.catalog]: This master's current role is: LEADER
I20260620 14:52:59.771821 22437 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 14:52:59.778901 22437 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 14:52:59.787740 22437 catalog_manager.cc:1347] Generated new cluster ID: 3798c804acbb4cbd881ccecc2fa0b329
I20260620 14:52:59.787870 22437 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 14:52:59.821524 22437 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 14:52:59.822041 22437 catalog_manager.cc:1504] Loading token signing keys...
I20260620 14:52:59.833550 22437 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 6bf2c7936e3c4602b461d7a128022b8b: Generated new TSK 0
I20260620 14:52:59.834530 22437 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260620 14:52:59.844388 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskUci3DU/build/asan/bin/kudu pid 22353
W20260620 14:52:59.845598 22415 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='slave'} at 127.0.0.1:38991
W20260620 14:52:59.846459 388 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskUci3DU/build/asan/bin/kudu pid 22353: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260620 14:52:59.846649 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskUci3DU/build/asan/bin/kudu with pid 22353
2026-06-20T14:52:59Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/14 (1952 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/15
Loading random data
Initializing database '/tmp/dist-test-taskUci3DU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1781966829091300-388-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 20 14:52:59 dist-test-slave-q9j6 krb5kdc[22451](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 20 14:52:59 dist-test-slave-q9j6 krb5kdc[22451](info): set up 2 sockets
Jun 20 14:52:59 dist-test-slave-q9j6 krb5kdc[22451](info): commencing operation
krb5kdc: starting...
W20260620 14:53:00.968102 388 mini_kdc.cc:121] Time spent starting KDC: real 1.085s user 0.004s sys 0.025s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskUci3DU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1781966829091300-388-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskUci3DU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1781966829091300-388-0/krb5kdc/test-user.keytab.
Jun 20 14:53:01 dist-test-slave-q9j6 krb5kdc[22451](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781967181, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-20T14:53:01Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T14:53:01Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.97.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskUci3DU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1781966829091300-388-0/krb5kdc/kudu_127.0.97.62.keytab.
Entry for principal kudu/127.0.97.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskUci3DU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1781966829091300-388-0/krb5kdc/kudu_127.0.97.62.keytab.
I20260620 14:53:01.241375 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskUci3DU/build/asan/bin/kudu
/tmp/dist-test-taskUci3DU/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskUci3DU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1781966829091300-388-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskUci3DU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1781966829091300-388-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskUci3DU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1781966829091300-388-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskUci3DU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1781966829091300-388-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:36717
--webserver_interface=127.0.97.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskUci3DU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1781966829091300-388-0/krb5kdc/kudu_127.0.97.62.keytab
--principal=kudu/127.0.97.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:42349
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.97.62:36717
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskUci3DU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1781966829091300-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskUci3DU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1781966829091300-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskUci3DU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1781966829091300-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 14:53:01.596240 22464 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 14:53:01.596565 22464 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 14:53:01.596750 22464 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 14:53:01.604555 22464 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 14:53:01.604689 22464 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 14:53:01.604771 22464 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 14:53:01.604887 22464 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 14:53:01.604948 22464 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 14:53:01.615370 22464 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:42349
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskUci3DU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1781966829091300-388-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskUci3DU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1781966829091300-388-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.97.62:36717
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskUci3DU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1781966829091300-388-0/krb5kdc/kudu_127.0.97.62.keytab
--principal=kudu/127.0.97.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:36717
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskUci3DU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1781966829091300-388-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.97.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskUci3DU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1781966829091300-388-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 13:41:20 UTC on ccycloud.quasar-vdagoi.root.comops.site
build id 4457462
ASAN enabled
I20260620 14:53:01.616808 22464 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 14:53:01.618284 22464 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 14:53:01.630963 22472 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 14:53:01.631031 22470 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 14:53:01.631186 22469 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 14:53:01.631855 22464 server_base.cc:1034] running on GCE node
I20260620 14:53:01.632627 22464 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 14:53:01.634099 22464 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 14:53:01.635308 22464 hybrid_clock.cc:648] HybridClock initialized: now 1781967181635246 us; error 55 us; skew 500 ppm
I20260620 14:53:01.635742 22464 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 14:53:01 dist-test-slave-q9j6 krb5kdc[22451](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781967181, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 14:53:01.639086 22464 init.cc:377] Logged in from keytab as kudu/127.0.97.62@KRBTEST.COM (short username kudu)
I20260620 14:53:01.640858 22464 webserver.cc:460] Webserver started at http://127.0.97.62:33147/ using document root <none> and password file <none>
I20260620 14:53:01.641633 22464 fs_manager.cc:359] Metadata directory not provided
I20260620 14:53:01.641775 22464 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 14:53:01.642132 22464 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 14:53:01.645449 22464 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskUci3DU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1781966829091300-388-0/minicluster-data/master-0/data/instance:
uuid: "8f6d7d9ccddb43099eb55943495837ba"
format_stamp: "Formatted at 2026-06-20 14:53:01 on dist-test-slave-q9j6"
server_key: "bacb9ac0da26779a4fc100c7441fe263"
server_key_iv: "378097d1946d8b86c4d8f362a06bdb43"
server_key_version: "encryptionkey@0"
I20260620 14:53:01.646293 22464 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskUci3DU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1781966829091300-388-0/minicluster-data/master-0/wal/instance:
uuid: "8f6d7d9ccddb43099eb55943495837ba"
format_stamp: "Formatted at 2026-06-20 14:53:01 on dist-test-slave-q9j6"
server_key: "bacb9ac0da26779a4fc100c7441fe263"
server_key_iv: "378097d1946d8b86c4d8f362a06bdb43"
server_key_version: "encryptionkey@0"
I20260620 14:53:01.652316 22464 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.007s sys 0.000s
I20260620 14:53:01.656471 22479 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 14:53:01.658071 22464 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260620 14:53:01.658314 22464 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskUci3DU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1781966829091300-388-0/minicluster-data/master-0/data,/tmp/dist-test-taskUci3DU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1781966829091300-388-0/minicluster-data/master-0/wal
uuid: "8f6d7d9ccddb43099eb55943495837ba"
format_stamp: "Formatted at 2026-06-20 14:53:01 on dist-test-slave-q9j6"
server_key: "bacb9ac0da26779a4fc100c7441fe263"
server_key_iv: "378097d1946d8b86c4d8f362a06bdb43"
server_key_version: "encryptionkey@0"
I20260620 14:53:01.658566 22464 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskUci3DU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1781966829091300-388-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskUci3DU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1781966829091300-388-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskUci3DU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1781966829091300-388-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 14:53:01.686764 22464 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 14:53:01.691361 22464 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 14:53:01.691762 22464 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 14:53:01.712795 22464 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.62:36717
I20260620 14:53:01.712836 22530 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.62:36717 every 8 connection(s)
I20260620 14:53:01.714437 22464 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskUci3DU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1781966829091300-388-0/minicluster-data/master-0/data/info.pb
I20260620 14:53:01.718997 22531 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 14:53:01.720999 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskUci3DU/build/asan/bin/kudu as pid 22464
I20260620 14:53:01.721221 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskUci3DU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1781966829091300-388-0/minicluster-data/master-0/wal/instance
I20260620 14:53:01.721716 388 external_mini_cluster.cc:1428] Setting key 90e1b0eaf00c5db065eb2aed6e35c849
I20260620 14:53:01.733711 22531 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 8f6d7d9ccddb43099eb55943495837ba: Bootstrap starting.
I20260620 14:53:01.737753 22531 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 8f6d7d9ccddb43099eb55943495837ba: Neither blocks nor log segments found. Creating new log.
Jun 20 14:53:01 dist-test-slave-q9j6 krb5kdc[22451](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781967181, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260620 14:53:01.738937 22531 log.cc:824] T 00000000000000000000000000000000 P 8f6d7d9ccddb43099eb55943495837ba: Log is configured to *not* fsync() on all Append() calls
I20260620 14:53:01.742889 22531 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 8f6d7d9ccddb43099eb55943495837ba: No bootstrap required, opened a new log
I20260620 14:53:01.749498 22534 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:53:01.724075 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39165 (local address 127.0.97.62:36717)
0620 14:53:01.725126 (+ 1051us) server_negotiation.cc:206] Beginning negotiation
0620 14:53:01.725151 (+ 25us) server_negotiation.cc:399] Waiting for connection header
0620 14:53:01.725210 (+ 59us) server_negotiation.cc:407] Connection header received
0620 14:53:01.727764 (+ 2554us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:53:01.727798 (+ 34us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:53:01.728415 (+ 617us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:53:01.728923 (+ 508us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:53:01.730549 (+ 1626us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:53:01.732694 (+ 2145us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:53:01.734258 (+ 1564us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:53:01.735333 (+ 1075us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:53:01.735576 (+ 243us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:53:01.739929 (+ 4353us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 14:53:01.739938 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 14:53:01.739959 (+ 21us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 14:53:01.740003 (+ 44us) server_negotiation.cc:922] calling sasl_server_start()
0620 14:53:01.744191 (+ 4188us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:53:01.745041 (+ 850us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:53:01.745051 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:53:01.745061 (+ 10us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:53:01.745305 (+ 244us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:53:01.745741 (+ 436us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:53:01.745746 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:53:01.745750 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:53:01.746571 (+ 821us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 14:53:01.746926 (+ 355us) server_negotiation.cc:1035] Waiting for connection context
0620 14:53:01.747327 (+ 401us) server_negotiation.cc:299] Negotiation successful
0620 14:53:01.747909 (+ 582us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":759,"thread_start_us":488,"threads_started":1}
I20260620 14:53:01.753430 22531 raft_consensus.cc:348] T 00000000000000000000000000000000 P 8f6d7d9ccddb43099eb55943495837ba [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8f6d7d9ccddb43099eb55943495837ba" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 36717 } }
I20260620 14:53:01.753880 22531 raft_consensus.cc:374] T 00000000000000000000000000000000 P 8f6d7d9ccddb43099eb55943495837ba [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 14:53:01.754021 22531 raft_consensus.cc:729] T 00000000000000000000000000000000 P 8f6d7d9ccddb43099eb55943495837ba [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8f6d7d9ccddb43099eb55943495837ba, State: Initialized, Role: FOLLOWER
I20260620 14:53:01.754331 388 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260620 14:53:01.754760 22531 consensus_queue.cc:260] T 00000000000000000000000000000000 P 8f6d7d9ccddb43099eb55943495837ba [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8f6d7d9ccddb43099eb55943495837ba" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 36717 } }
I20260620 14:53:01.755021 22531 raft_consensus.cc:388] T 00000000000000000000000000000000 P 8f6d7d9ccddb43099eb55943495837ba [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 14:53:01.755188 22531 raft_consensus.cc:482] T 00000000000000000000000000000000 P 8f6d7d9ccddb43099eb55943495837ba [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 14:53:01.755366 22531 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 8f6d7d9ccddb43099eb55943495837ba [term 0 FOLLOWER]: Advancing to term 1
I20260620 14:53:01.757761 22531 raft_consensus.cc:504] T 00000000000000000000000000000000 P 8f6d7d9ccddb43099eb55943495837ba [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8f6d7d9ccddb43099eb55943495837ba" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 36717 } }
I20260620 14:53:01.758414 22531 leader_election.cc:302] T 00000000000000000000000000000000 P 8f6d7d9ccddb43099eb55943495837ba [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 8f6d7d9ccddb43099eb55943495837ba; no voters:
I20260620 14:53:01.759204 22531 leader_election.cc:288] T 00000000000000000000000000000000 P 8f6d7d9ccddb43099eb55943495837ba [CANDIDATE]: Term 1 election: Requested vote from peers
I20260620 14:53:01.759315 22540 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 8f6d7d9ccddb43099eb55943495837ba [term 1 FOLLOWER]: Leader election won for term 1
I20260620 14:53:01.760710 22540 raft_consensus.cc:686] T 00000000000000000000000000000000 P 8f6d7d9ccddb43099eb55943495837ba [term 1 LEADER]: Becoming Leader. State: Replica: 8f6d7d9ccddb43099eb55943495837ba, State: Running, Role: LEADER
I20260620 14:53:01.761543 22540 consensus_queue.cc:237] T 00000000000000000000000000000000 P 8f6d7d9ccddb43099eb55943495837ba [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8f6d7d9ccddb43099eb55943495837ba" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 36717 } }
I20260620 14:53:01.763489 22531 sys_catalog.cc:564] T 00000000000000000000000000000000 P 8f6d7d9ccddb43099eb55943495837ba [sys.catalog]: configured and running, proceeding with master startup.
I20260620 14:53:01.764619 22543 sys_catalog.cc:455] T 00000000000000000000000000000000 P 8f6d7d9ccddb43099eb55943495837ba [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "8f6d7d9ccddb43099eb55943495837ba" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8f6d7d9ccddb43099eb55943495837ba" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 36717 } } }
I20260620 14:53:01.765072 22543 sys_catalog.cc:458] T 00000000000000000000000000000000 P 8f6d7d9ccddb43099eb55943495837ba [sys.catalog]: This master's current role is: LEADER
I20260620 14:53:01.768515 22542 sys_catalog.cc:455] T 00000000000000000000000000000000 P 8f6d7d9ccddb43099eb55943495837ba [sys.catalog]: SysCatalogTable state changed. Reason: New leader 8f6d7d9ccddb43099eb55943495837ba. Latest consensus state: current_term: 1 leader_uuid: "8f6d7d9ccddb43099eb55943495837ba" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8f6d7d9ccddb43099eb55943495837ba" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 36717 } } }
I20260620 14:53:01.769078 22542 sys_catalog.cc:458] T 00000000000000000000000000000000 P 8f6d7d9ccddb43099eb55943495837ba [sys.catalog]: This master's current role is: LEADER
I20260620 14:53:01.772028 22546 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 14:53:01.778264 22546 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 14:53:01.782712 22534 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:53:01.759168 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46133 (local address 127.0.97.62:36717)
0620 14:53:01.759381 (+ 213us) server_negotiation.cc:206] Beginning negotiation
0620 14:53:01.759393 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0620 14:53:01.760161 (+ 768us) server_negotiation.cc:407] Connection header received
0620 14:53:01.760437 (+ 276us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:53:01.760446 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:53:01.760586 (+ 140us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:53:01.760758 (+ 172us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:53:01.763242 (+ 2484us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:53:01.764645 (+ 1403us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:53:01.767204 (+ 2559us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:53:01.768203 (+ 999us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:53:01.768383 (+ 180us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:53:01.772821 (+ 4438us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 14:53:01.772831 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 14:53:01.772836 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 14:53:01.772894 (+ 58us) server_negotiation.cc:922] calling sasl_server_start()
0620 14:53:01.778208 (+ 5314us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:53:01.779574 (+ 1366us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:53:01.779583 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:53:01.779590 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:53:01.779870 (+ 280us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:53:01.780520 (+ 650us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:53:01.780532 (+ 12us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:53:01.780538 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:53:01.781223 (+ 685us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 14:53:01.781493 (+ 270us) server_negotiation.cc:1035] Waiting for connection context
0620 14:53:01.782072 (+ 579us) server_negotiation.cc:299] Negotiation successful
0620 14:53:01.782326 (+ 254us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":67}
I20260620 14:53:01.790958 22546 catalog_manager.cc:1347] Generated new cluster ID: 0baf84dff0284bf0ad5672fae5cb7a8b
I20260620 14:53:01.791227 22546 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 14:53:01.807674 22546 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 14:53:01.809180 22546 catalog_manager.cc:1504] Loading token signing keys...
I20260620 14:53:01.818755 22546 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 8f6d7d9ccddb43099eb55943495837ba: Generated new TSK 0
I20260620 14:53:01.820030 22546 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260620 14:53:01.827996 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskUci3DU/build/asan/bin/kudu pid 22464
I20260620 14:53:01.829816 22526 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 14:53:01.922749 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskUci3DU/build/asan/bin/kudu with pid 22464
2026-06-20T14:53:01Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/15 (2077 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/16
Loading random data
Initializing database '/tmp/dist-test-taskUci3DU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1781966829091300-388-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 20 14:53:02 dist-test-slave-q9j6 krb5kdc[22563](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 20 14:53:02 dist-test-slave-q9j6 krb5kdc[22563](info): set up 2 sockets
Jun 20 14:53:02 dist-test-slave-q9j6 krb5kdc[22563](info): commencing operation
krb5kdc: starting...
W20260620 14:53:03.032196 388 mini_kdc.cc:121] Time spent starting KDC: real 1.071s user 0.010s sys 0.020s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskUci3DU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1781966829091300-388-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskUci3DU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1781966829091300-388-0/krb5kdc/test-user.keytab.
Jun 20 14:53:03 dist-test-slave-q9j6 krb5kdc[22563](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781967183, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-20T14:53:03Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T14:53:03Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.97.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskUci3DU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1781966829091300-388-0/krb5kdc/kudu_127.0.97.62.keytab.
Entry for principal kudu/127.0.97.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskUci3DU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1781966829091300-388-0/krb5kdc/kudu_127.0.97.62.keytab.
I20260620 14:53:03.311928 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskUci3DU/build/asan/bin/kudu
/tmp/dist-test-taskUci3DU/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskUci3DU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1781966829091300-388-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskUci3DU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1781966829091300-388-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskUci3DU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1781966829091300-388-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskUci3DU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1781966829091300-388-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:36069
--webserver_interface=127.0.97.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskUci3DU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1781966829091300-388-0/krb5kdc/kudu_127.0.97.62.keytab
--principal=kudu/127.0.97.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:45649
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.97.62:36069
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskUci3DU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1781966829091300-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskUci3DU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1781966829091300-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskUci3DU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1781966829091300-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 14:53:03.659883 22576 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 14:53:03.660205 22576 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 14:53:03.660431 22576 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 14:53:03.668076 22576 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 14:53:03.668244 22576 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 14:53:03.668341 22576 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 14:53:03.668452 22576 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 14:53:03.668524 22576 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 14:53:03.678849 22576 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:45649
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskUci3DU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1781966829091300-388-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskUci3DU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1781966829091300-388-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.97.62:36069
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskUci3DU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1781966829091300-388-0/krb5kdc/kudu_127.0.97.62.keytab
--principal=kudu/127.0.97.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:36069
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskUci3DU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1781966829091300-388-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.97.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskUci3DU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1781966829091300-388-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 13:41:20 UTC on ccycloud.quasar-vdagoi.root.comops.site
build id 4457462
ASAN enabled
I20260620 14:53:03.680333 22576 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 14:53:03.681753 22576 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 14:53:03.694509 22581 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 14:53:03.694398 22584 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 14:53:03.694446 22582 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 14:53:03.695325 22576 server_base.cc:1034] running on GCE node
I20260620 14:53:03.696221 22576 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 14:53:03.697738 22576 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 14:53:03.699008 22576 hybrid_clock.cc:648] HybridClock initialized: now 1781967183698941 us; error 60 us; skew 500 ppm
I20260620 14:53:03.699450 22576 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 14:53:03 dist-test-slave-q9j6 krb5kdc[22563](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781967183, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 14:53:03.706359 22576 init.cc:377] Logged in from keytab as kudu/127.0.97.62@KRBTEST.COM (short username kudu)
I20260620 14:53:03.708194 22576 webserver.cc:460] Webserver started at http://127.0.97.62:46013/ using document root <none> and password file <none>
I20260620 14:53:03.708935 22576 fs_manager.cc:359] Metadata directory not provided
I20260620 14:53:03.709050 22576 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 14:53:03.709347 22576 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 14:53:03.712734 22576 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskUci3DU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1781966829091300-388-0/minicluster-data/master-0/data/instance:
uuid: "d7d1464334564ddd8d81d5155bcc0a2b"
format_stamp: "Formatted at 2026-06-20 14:53:03 on dist-test-slave-q9j6"
server_key: "0da3c62789dea3f4185d81fec2a0ab6d"
server_key_iv: "a0aa37ffd6a9747a2106b0c1b2341d4c"
server_key_version: "encryptionkey@0"
I20260620 14:53:03.713541 22576 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskUci3DU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1781966829091300-388-0/minicluster-data/master-0/wal/instance:
uuid: "d7d1464334564ddd8d81d5155bcc0a2b"
format_stamp: "Formatted at 2026-06-20 14:53:03 on dist-test-slave-q9j6"
server_key: "0da3c62789dea3f4185d81fec2a0ab6d"
server_key_iv: "a0aa37ffd6a9747a2106b0c1b2341d4c"
server_key_version: "encryptionkey@0"
I20260620 14:53:03.719215 22576 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.000s
I20260620 14:53:03.723133 22591 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 14:53:03.724882 22576 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260620 14:53:03.725072 22576 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskUci3DU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1781966829091300-388-0/minicluster-data/master-0/data,/tmp/dist-test-taskUci3DU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1781966829091300-388-0/minicluster-data/master-0/wal
uuid: "d7d1464334564ddd8d81d5155bcc0a2b"
format_stamp: "Formatted at 2026-06-20 14:53:03 on dist-test-slave-q9j6"
server_key: "0da3c62789dea3f4185d81fec2a0ab6d"
server_key_iv: "a0aa37ffd6a9747a2106b0c1b2341d4c"
server_key_version: "encryptionkey@0"
I20260620 14:53:03.725281 22576 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskUci3DU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1781966829091300-388-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskUci3DU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1781966829091300-388-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskUci3DU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1781966829091300-388-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 14:53:03.753334 22576 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 14:53:03.758219 22576 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 14:53:03.758479 22576 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 14:53:03.778360 22576 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.62:36069
I20260620 14:53:03.778415 22642 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.62:36069 every 8 connection(s)
I20260620 14:53:03.780011 22576 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskUci3DU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1781966829091300-388-0/minicluster-data/master-0/data/info.pb
I20260620 14:53:03.784227 22643 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 14:53:03.789047 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskUci3DU/build/asan/bin/kudu as pid 22576
I20260620 14:53:03.789251 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskUci3DU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1781966829091300-388-0/minicluster-data/master-0/wal/instance
I20260620 14:53:03.789813 388 external_mini_cluster.cc:1428] Setting key 2789ec0da3f489de3277abd4e88a8147
I20260620 14:53:03.797258 22643 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d7d1464334564ddd8d81d5155bcc0a2b: Bootstrap starting.
I20260620 14:53:03.801308 22643 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P d7d1464334564ddd8d81d5155bcc0a2b: Neither blocks nor log segments found. Creating new log.
I20260620 14:53:03.802752 22643 log.cc:824] T 00000000000000000000000000000000 P d7d1464334564ddd8d81d5155bcc0a2b: Log is configured to *not* fsync() on all Append() calls
I20260620 14:53:03.805857 22646 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:53:03.791947 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57223 (local address 127.0.97.62:36069)
0620 14:53:03.792842 (+ 895us) server_negotiation.cc:206] Beginning negotiation
0620 14:53:03.792857 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0620 14:53:03.792903 (+ 46us) server_negotiation.cc:407] Connection header received
0620 14:53:03.795376 (+ 2473us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:53:03.795407 (+ 31us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:53:03.796007 (+ 600us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:53:03.796392 (+ 385us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:53:03.797090 (+ 698us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:53:03.798560 (+ 1470us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:53:03.800574 (+ 2014us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:53:03.801318 (+ 744us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:53:03.801556 (+ 238us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:53:03.802852 (+ 1296us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 14:53:03.802888 (+ 36us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 14:53:03.802910 (+ 22us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 14:53:03.803708 (+ 798us) server_negotiation.cc:922] calling sasl_server_start()
0620 14:53:03.803820 (+ 112us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 14:53:03.803858 (+ 38us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 14:53:03.804002 (+ 144us) server_negotiation.cc:1035] Waiting for connection context
0620 14:53:03.804235 (+ 233us) server_negotiation.cc:299] Negotiation successful
0620 14:53:03.804537 (+ 302us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":657,"thread_start_us":432,"threads_started":1}
I20260620 14:53:03.806519 22643 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d7d1464334564ddd8d81d5155bcc0a2b: No bootstrap required, opened a new log
I20260620 14:53:03.810098 388 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260620 14:53:03.815042 22643 raft_consensus.cc:348] T 00000000000000000000000000000000 P d7d1464334564ddd8d81d5155bcc0a2b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d7d1464334564ddd8d81d5155bcc0a2b" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 36069 } }
I20260620 14:53:03.815634 22643 raft_consensus.cc:374] T 00000000000000000000000000000000 P d7d1464334564ddd8d81d5155bcc0a2b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 14:53:03.815766 22643 raft_consensus.cc:729] T 00000000000000000000000000000000 P d7d1464334564ddd8d81d5155bcc0a2b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d7d1464334564ddd8d81d5155bcc0a2b, State: Initialized, Role: FOLLOWER
I20260620 14:53:03.816478 22643 consensus_queue.cc:260] T 00000000000000000000000000000000 P d7d1464334564ddd8d81d5155bcc0a2b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d7d1464334564ddd8d81d5155bcc0a2b" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 36069 } }
I20260620 14:53:03.816749 22643 raft_consensus.cc:388] T 00000000000000000000000000000000 P d7d1464334564ddd8d81d5155bcc0a2b [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 14:53:03.816919 22643 raft_consensus.cc:482] T 00000000000000000000000000000000 P d7d1464334564ddd8d81d5155bcc0a2b [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 14:53:03.817111 22643 raft_consensus.cc:3037] T 00000000000000000000000000000000 P d7d1464334564ddd8d81d5155bcc0a2b [term 0 FOLLOWER]: Advancing to term 1
I20260620 14:53:03.819480 22643 raft_consensus.cc:504] T 00000000000000000000000000000000 P d7d1464334564ddd8d81d5155bcc0a2b [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d7d1464334564ddd8d81d5155bcc0a2b" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 36069 } }
I20260620 14:53:03.820122 22643 leader_election.cc:302] T 00000000000000000000000000000000 P d7d1464334564ddd8d81d5155bcc0a2b [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: d7d1464334564ddd8d81d5155bcc0a2b; no voters:
I20260620 14:53:03.820909 22643 leader_election.cc:288] T 00000000000000000000000000000000 P d7d1464334564ddd8d81d5155bcc0a2b [CANDIDATE]: Term 1 election: Requested vote from peers
I20260620 14:53:03.821053 22653 raft_consensus.cc:2781] T 00000000000000000000000000000000 P d7d1464334564ddd8d81d5155bcc0a2b [term 1 FOLLOWER]: Leader election won for term 1
I20260620 14:53:03.821312 22646 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:53:03.814676 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55595 (local address 127.0.97.62:36069)
0620 14:53:03.814869 (+ 193us) server_negotiation.cc:206] Beginning negotiation
0620 14:53:03.814876 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0620 14:53:03.815234 (+ 358us) server_negotiation.cc:407] Connection header received
0620 14:53:03.815391 (+ 157us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:53:03.815396 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:53:03.815492 (+ 96us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:53:03.815639 (+ 147us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:53:03.816252 (+ 613us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:53:03.817106 (+ 854us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:53:03.818476 (+ 1370us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:53:03.819017 (+ 541us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:53:03.819174 (+ 157us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:53:03.820361 (+ 1187us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 14:53:03.820373 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 14:53:03.820377 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 14:53:03.820397 (+ 20us) server_negotiation.cc:922] calling sasl_server_start()
0620 14:53:03.820447 (+ 50us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 14:53:03.820464 (+ 17us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 14:53:03.820564 (+ 100us) server_negotiation.cc:1035] Waiting for connection context
0620 14:53:03.820712 (+ 148us) server_negotiation.cc:299] Negotiation successful
0620 14:53:03.820814 (+ 102us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":73}
I20260620 14:53:03.824074 22643 sys_catalog.cc:564] T 00000000000000000000000000000000 P d7d1464334564ddd8d81d5155bcc0a2b [sys.catalog]: configured and running, proceeding with master startup.
I20260620 14:53:03.828068 22653 raft_consensus.cc:686] T 00000000000000000000000000000000 P d7d1464334564ddd8d81d5155bcc0a2b [term 1 LEADER]: Becoming Leader. State: Replica: d7d1464334564ddd8d81d5155bcc0a2b, State: Running, Role: LEADER
I20260620 14:53:03.828995 22653 consensus_queue.cc:237] T 00000000000000000000000000000000 P d7d1464334564ddd8d81d5155bcc0a2b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d7d1464334564ddd8d81d5155bcc0a2b" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 36069 } }
I20260620 14:53:03.833067 22654 sys_catalog.cc:455] T 00000000000000000000000000000000 P d7d1464334564ddd8d81d5155bcc0a2b [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "d7d1464334564ddd8d81d5155bcc0a2b" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d7d1464334564ddd8d81d5155bcc0a2b" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 36069 } } }
I20260620 14:53:03.833120 22655 sys_catalog.cc:455] T 00000000000000000000000000000000 P d7d1464334564ddd8d81d5155bcc0a2b [sys.catalog]: SysCatalogTable state changed. Reason: New leader d7d1464334564ddd8d81d5155bcc0a2b. Latest consensus state: current_term: 1 leader_uuid: "d7d1464334564ddd8d81d5155bcc0a2b" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d7d1464334564ddd8d81d5155bcc0a2b" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 36069 } } }
I20260620 14:53:03.833590 22655 sys_catalog.cc:458] T 00000000000000000000000000000000 P d7d1464334564ddd8d81d5155bcc0a2b [sys.catalog]: This master's current role is: LEADER
I20260620 14:53:03.833593 22654 sys_catalog.cc:458] T 00000000000000000000000000000000 P d7d1464334564ddd8d81d5155bcc0a2b [sys.catalog]: This master's current role is: LEADER
I20260620 14:53:03.835227 22667 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 14:53:03.840794 22667 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 14:53:03.850479 22667 catalog_manager.cc:1347] Generated new cluster ID: c5fa10a22fc64bd6a989cfe724abfee7
I20260620 14:53:03.850622 22667 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 14:53:03.871810 22667 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 14:53:03.873299 22667 catalog_manager.cc:1504] Loading token signing keys...
I20260620 14:53:03.880316 22667 catalog_manager.cc:5868] T 00000000000000000000000000000000 P d7d1464334564ddd8d81d5155bcc0a2b: Generated new TSK 0
I20260620 14:53:03.881301 22667 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260620 14:53:03.896988 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskUci3DU/build/asan/bin/kudu pid 22576
W20260620 14:53:03.898365 22638 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='slave'} at 127.0.0.1:57223
W20260620 14:53:03.899216 388 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskUci3DU/build/asan/bin/kudu pid 22576: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260620 14:53:03.899390 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskUci3DU/build/asan/bin/kudu with pid 22576
2026-06-20T14:53:03Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/16 (1975 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/17
Loading random data
Initializing database '/tmp/dist-test-taskUci3DU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1781966829091300-388-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 20 14:53:04 dist-test-slave-q9j6 krb5kdc[22674](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 20 14:53:04 dist-test-slave-q9j6 krb5kdc[22674](info): set up 2 sockets
Jun 20 14:53:04 dist-test-slave-q9j6 krb5kdc[22674](info): commencing operation
krb5kdc: starting...
W20260620 14:53:05.028162 388 mini_kdc.cc:121] Time spent starting KDC: real 1.092s user 0.004s sys 0.028s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskUci3DU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1781966829091300-388-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskUci3DU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1781966829091300-388-0/krb5kdc/test-user.keytab.
Jun 20 14:53:05 dist-test-slave-q9j6 krb5kdc[22674](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781967185, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-20T14:53:05Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T14:53:05Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.97.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskUci3DU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1781966829091300-388-0/krb5kdc/kudu_127.0.97.62.keytab.
Entry for principal kudu/127.0.97.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskUci3DU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1781966829091300-388-0/krb5kdc/kudu_127.0.97.62.keytab.
I20260620 14:53:05.307420 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskUci3DU/build/asan/bin/kudu
/tmp/dist-test-taskUci3DU/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskUci3DU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1781966829091300-388-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskUci3DU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1781966829091300-388-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskUci3DU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1781966829091300-388-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskUci3DU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1781966829091300-388-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:46233
--webserver_interface=127.0.97.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskUci3DU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1781966829091300-388-0/krb5kdc/kudu_127.0.97.62.keytab
--principal=kudu/127.0.97.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:43875
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.97.62:46233
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskUci3DU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1781966829091300-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskUci3DU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1781966829091300-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskUci3DU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1781966829091300-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 14:53:05.654039 22687 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 14:53:05.654363 22687 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 14:53:05.654560 22687 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 14:53:05.662137 22687 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 14:53:05.662276 22687 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 14:53:05.662408 22687 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 14:53:05.662530 22687 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 14:53:05.662612 22687 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 14:53:05.672936 22687 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:43875
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskUci3DU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1781966829091300-388-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskUci3DU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1781966829091300-388-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.97.62:46233
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskUci3DU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1781966829091300-388-0/krb5kdc/kudu_127.0.97.62.keytab
--principal=kudu/127.0.97.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:46233
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskUci3DU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1781966829091300-388-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.97.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskUci3DU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1781966829091300-388-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 13:41:20 UTC on ccycloud.quasar-vdagoi.root.comops.site
build id 4457462
ASAN enabled
I20260620 14:53:05.674397 22687 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 14:53:05.675830 22687 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 14:53:05.688701 22695 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 14:53:05.688786 22693 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 14:53:05.688805 22692 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 14:53:05.689604 22687 server_base.cc:1034] running on GCE node
I20260620 14:53:05.690490 22687 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 14:53:05.692176 22687 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 14:53:05.693445 22687 hybrid_clock.cc:648] HybridClock initialized: now 1781967185693391 us; error 50 us; skew 500 ppm
I20260620 14:53:05.693842 22687 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 14:53:05 dist-test-slave-q9j6 krb5kdc[22674](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781967185, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 14:53:05.697283 22687 init.cc:377] Logged in from keytab as kudu/127.0.97.62@KRBTEST.COM (short username kudu)
I20260620 14:53:05.698895 22687 webserver.cc:460] Webserver started at http://127.0.97.62:35551/ using document root <none> and password file <none>
I20260620 14:53:05.699733 22687 fs_manager.cc:359] Metadata directory not provided
I20260620 14:53:05.699847 22687 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 14:53:05.700150 22687 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 14:53:05.703435 22687 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskUci3DU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1781966829091300-388-0/minicluster-data/master-0/data/instance:
uuid: "5743fbb4e6b04a61885e48773c369726"
format_stamp: "Formatted at 2026-06-20 14:53:05 on dist-test-slave-q9j6"
server_key: "593a335b07c78b92e49323189eda6a83"
server_key_iv: "6f924e25323848fc6a77dcef83e12722"
server_key_version: "encryptionkey@0"
I20260620 14:53:05.704372 22687 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskUci3DU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1781966829091300-388-0/minicluster-data/master-0/wal/instance:
uuid: "5743fbb4e6b04a61885e48773c369726"
format_stamp: "Formatted at 2026-06-20 14:53:05 on dist-test-slave-q9j6"
server_key: "593a335b07c78b92e49323189eda6a83"
server_key_iv: "6f924e25323848fc6a77dcef83e12722"
server_key_version: "encryptionkey@0"
I20260620 14:53:05.710886 22687 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.001s sys 0.004s
I20260620 14:53:05.716456 22702 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 14:53:05.718344 22687 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.001s sys 0.004s
I20260620 14:53:05.718588 22687 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskUci3DU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1781966829091300-388-0/minicluster-data/master-0/data,/tmp/dist-test-taskUci3DU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1781966829091300-388-0/minicluster-data/master-0/wal
uuid: "5743fbb4e6b04a61885e48773c369726"
format_stamp: "Formatted at 2026-06-20 14:53:05 on dist-test-slave-q9j6"
server_key: "593a335b07c78b92e49323189eda6a83"
server_key_iv: "6f924e25323848fc6a77dcef83e12722"
server_key_version: "encryptionkey@0"
I20260620 14:53:05.718827 22687 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskUci3DU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1781966829091300-388-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskUci3DU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1781966829091300-388-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskUci3DU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1781966829091300-388-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 14:53:05.725250 22687 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 14:53:05.730087 22687 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 14:53:05.730432 22687 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 14:53:05.750250 22687 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.62:46233
I20260620 14:53:05.750286 22753 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.62:46233 every 8 connection(s)
I20260620 14:53:05.751868 22687 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskUci3DU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1781966829091300-388-0/minicluster-data/master-0/data/info.pb
I20260620 14:53:05.754657 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskUci3DU/build/asan/bin/kudu as pid 22687
I20260620 14:53:05.754846 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskUci3DU/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1781966829091300-388-0/minicluster-data/master-0/wal/instance
I20260620 14:53:05.755388 388 external_mini_cluster.cc:1428] Setting key 731019712deda1b8ceb90932b4f040a9
I20260620 14:53:05.756595 22754 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 14:53:05.764393 22757 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:53:05.757389 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:32933 (local address 127.0.97.62:46233)
0620 14:53:05.758357 (+ 968us) server_negotiation.cc:206] Beginning negotiation
0620 14:53:05.758372 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0620 14:53:05.758423 (+ 51us) server_negotiation.cc:407] Connection header received
0620 14:53:05.760269 (+ 1846us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:53:05.760297 (+ 28us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:53:05.760753 (+ 456us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:53:05.761176 (+ 423us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:53:05.762373 (+ 1197us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 14:53:05.762380 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 14:53:05.762399 (+ 19us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 14:53:05.762421 (+ 22us) server_negotiation.cc:922] calling sasl_server_start()
0620 14:53:05.762497 (+ 76us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 14:53:05.762519 (+ 22us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 14:53:05.762720 (+ 201us) server_negotiation.cc:1035] Waiting for connection context
0620 14:53:05.762838 (+ 118us) server_negotiation.cc:299] Negotiation successful
0620 14:53:05.763116 (+ 278us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":681,"thread_start_us":427,"threads_started":1}
I20260620 14:53:05.768872 388 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260620 14:53:05.771071 22754 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 5743fbb4e6b04a61885e48773c369726: Bootstrap starting.
I20260620 14:53:05.774873 22754 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 5743fbb4e6b04a61885e48773c369726: Neither blocks nor log segments found. Creating new log.
I20260620 14:53:05.776151 22757 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:53:05.773312 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51945 (local address 127.0.97.62:46233)
0620 14:53:05.773513 (+ 201us) server_negotiation.cc:206] Beginning negotiation
0620 14:53:05.773520 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0620 14:53:05.774035 (+ 515us) server_negotiation.cc:407] Connection header received
0620 14:53:05.774187 (+ 152us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:53:05.774193 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:53:05.774257 (+ 64us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:53:05.774373 (+ 116us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:53:05.775363 (+ 990us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 14:53:05.775371 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 14:53:05.775377 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 14:53:05.775404 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0620 14:53:05.775456 (+ 52us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 14:53:05.775476 (+ 20us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 14:53:05.775619 (+ 143us) server_negotiation.cc:1035] Waiting for connection context
0620 14:53:05.775762 (+ 143us) server_negotiation.cc:299] Negotiation successful
0620 14:53:05.775825 (+ 63us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":68}
I20260620 14:53:05.776388 22754 log.cc:824] T 00000000000000000000000000000000 P 5743fbb4e6b04a61885e48773c369726: Log is configured to *not* fsync() on all Append() calls
I20260620 14:53:05.779867 22754 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 5743fbb4e6b04a61885e48773c369726: No bootstrap required, opened a new log
I20260620 14:53:05.787845 22754 raft_consensus.cc:348] T 00000000000000000000000000000000 P 5743fbb4e6b04a61885e48773c369726 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5743fbb4e6b04a61885e48773c369726" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 46233 } }
I20260620 14:53:05.788290 22754 raft_consensus.cc:374] T 00000000000000000000000000000000 P 5743fbb4e6b04a61885e48773c369726 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 14:53:05.788439 22754 raft_consensus.cc:729] T 00000000000000000000000000000000 P 5743fbb4e6b04a61885e48773c369726 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5743fbb4e6b04a61885e48773c369726, State: Initialized, Role: FOLLOWER
I20260620 14:53:05.789110 22754 consensus_queue.cc:260] T 00000000000000000000000000000000 P 5743fbb4e6b04a61885e48773c369726 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5743fbb4e6b04a61885e48773c369726" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 46233 } }
I20260620 14:53:05.789422 22754 raft_consensus.cc:388] T 00000000000000000000000000000000 P 5743fbb4e6b04a61885e48773c369726 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 14:53:05.789588 22754 raft_consensus.cc:482] T 00000000000000000000000000000000 P 5743fbb4e6b04a61885e48773c369726 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 14:53:05.789778 22754 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 5743fbb4e6b04a61885e48773c369726 [term 0 FOLLOWER]: Advancing to term 1
I20260620 14:53:05.792138 22754 raft_consensus.cc:504] T 00000000000000000000000000000000 P 5743fbb4e6b04a61885e48773c369726 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5743fbb4e6b04a61885e48773c369726" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 46233 } }
I20260620 14:53:05.792671 22754 leader_election.cc:302] T 00000000000000000000000000000000 P 5743fbb4e6b04a61885e48773c369726 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 5743fbb4e6b04a61885e48773c369726; no voters:
I20260620 14:53:05.793277 22754 leader_election.cc:288] T 00000000000000000000000000000000 P 5743fbb4e6b04a61885e48773c369726 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260620 14:53:05.793550 22764 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 5743fbb4e6b04a61885e48773c369726 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 14:53:05.794281 22764 raft_consensus.cc:686] T 00000000000000000000000000000000 P 5743fbb4e6b04a61885e48773c369726 [term 1 LEADER]: Becoming Leader. State: Replica: 5743fbb4e6b04a61885e48773c369726, State: Running, Role: LEADER
I20260620 14:53:05.794838 22764 consensus_queue.cc:237] T 00000000000000000000000000000000 P 5743fbb4e6b04a61885e48773c369726 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5743fbb4e6b04a61885e48773c369726" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 46233 } }
I20260620 14:53:05.795434 22754 sys_catalog.cc:564] T 00000000000000000000000000000000 P 5743fbb4e6b04a61885e48773c369726 [sys.catalog]: configured and running, proceeding with master startup.
I20260620 14:53:05.797664 22766 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5743fbb4e6b04a61885e48773c369726 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 5743fbb4e6b04a61885e48773c369726. Latest consensus state: current_term: 1 leader_uuid: "5743fbb4e6b04a61885e48773c369726" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5743fbb4e6b04a61885e48773c369726" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 46233 } } }
I20260620 14:53:05.798198 22766 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5743fbb4e6b04a61885e48773c369726 [sys.catalog]: This master's current role is: LEADER
I20260620 14:53:05.798113 22765 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5743fbb4e6b04a61885e48773c369726 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "5743fbb4e6b04a61885e48773c369726" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5743fbb4e6b04a61885e48773c369726" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 46233 } } }
I20260620 14:53:05.798385 22765 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5743fbb4e6b04a61885e48773c369726 [sys.catalog]: This master's current role is: LEADER
I20260620 14:53:05.800671 22770 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 14:53:05.806244 22770 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 14:53:05.816278 22770 catalog_manager.cc:1347] Generated new cluster ID: 456b585e39a04145958cd683c844f3f7
I20260620 14:53:05.816383 22770 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 14:53:05.846944 22770 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 14:53:05.847434 22770 catalog_manager.cc:1504] Loading token signing keys...
I20260620 14:53:05.857955 22770 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 5743fbb4e6b04a61885e48773c369726: Generated new TSK 0
I20260620 14:53:05.858893 22770 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260620 14:53:05.873817 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskUci3DU/build/asan/bin/kudu pid 22687
W20260620 14:53:05.875888 22749 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='slave'} at 127.0.0.1:32933
W20260620 14:53:05.876636 388 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskUci3DU/build/asan/bin/kudu pid 22687: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260620 14:53:05.876788 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskUci3DU/build/asan/bin/kudu with pid 22687
2026-06-20T14:53:05Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/17 (1976 ms)
[----------] 18 tests from AuthTokenIssuingTest (36231 ms total)
[----------] 2 tests from ConnectToFollowerMasterTest
[ RUN ] ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys/0
Loading random data
Initializing database '/tmp/dist-test-taskUci3DU/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781966829091300-388-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 20 14:53:05 dist-test-slave-q9j6 krb5kdc[22785](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 20 14:53:05 dist-test-slave-q9j6 krb5kdc[22785](info): set up 2 sockets
Jun 20 14:53:05 dist-test-slave-q9j6 krb5kdc[22785](info): commencing operation
krb5kdc: starting...
W20260620 14:53:06.992249 388 mini_kdc.cc:121] Time spent starting KDC: real 1.078s user 0.007s sys 0.022s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskUci3DU/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781966829091300-388-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskUci3DU/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781966829091300-388-0/krb5kdc/test-user.keytab.
Jun 20 14:53:07 dist-test-slave-q9j6 krb5kdc[22785](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781967187, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-20T14:53:07Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T14:53:07Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.97.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskUci3DU/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781966829091300-388-0/krb5kdc/kudu_127.0.97.62.keytab.
Entry for principal kudu/127.0.97.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskUci3DU/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781966829091300-388-0/krb5kdc/kudu_127.0.97.62.keytab.
I20260620 14:53:07.263104 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskUci3DU/build/asan/bin/kudu
/tmp/dist-test-taskUci3DU/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskUci3DU/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781966829091300-388-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskUci3DU/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781966829091300-388-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskUci3DU/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781966829091300-388-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskUci3DU/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781966829091300-388-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:45139
--webserver_interface=127.0.97.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskUci3DU/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781966829091300-388-0/krb5kdc/kudu_127.0.97.62.keytab
--principal=kudu/127.0.97.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:37801
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.97.62:45139,127.0.97.61:45529,127.0.97.60:33721
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=optional with env {KRB5CCNAME=/tmp/dist-test-taskUci3DU/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781966829091300-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskUci3DU/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781966829091300-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskUci3DU/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781966829091300-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 14:53:07.631448 22798 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260620 14:53:07.631919 22798 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 14:53:07.632051 22798 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 14:53:07.632190 22798 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 14:53:07.639835 22798 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 14:53:07.639997 22798 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 14:53:07.640178 22798 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 14:53:07.640357 22798 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 14:53:07.640502 22798 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 14:53:07.650853 22798 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:37801
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-taskUci3DU/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781966829091300-388-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskUci3DU/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781966829091300-388-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.97.62:45139,127.0.97.61:45529,127.0.97.60:33721
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskUci3DU/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781966829091300-388-0/krb5kdc/kudu_127.0.97.62.keytab
--principal=kudu/127.0.97.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:45139
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskUci3DU/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781966829091300-388-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.97.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskUci3DU/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781966829091300-388-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 13:41:20 UTC on ccycloud.quasar-vdagoi.root.comops.site
build id 4457462
ASAN enabled
I20260620 14:53:07.652338 22798 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 14:53:07.653792 22798 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 14:53:07.666687 22803 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 14:53:07.666613 22806 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 14:53:07.666658 22804 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 14:53:07.667577 22798 server_base.cc:1034] running on GCE node
I20260620 14:53:07.668524 22798 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 14:53:07.670141 22798 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 14:53:07.671406 22798 hybrid_clock.cc:648] HybridClock initialized: now 1781967187671329 us; error 62 us; skew 500 ppm
I20260620 14:53:07.671829 22798 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 14:53:07 dist-test-slave-q9j6 krb5kdc[22785](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781967187, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 14:53:07.675177 22798 init.cc:377] Logged in from keytab as kudu/127.0.97.62@KRBTEST.COM (short username kudu)
I20260620 14:53:07.676936 22798 webserver.cc:460] Webserver started at http://127.0.97.62:36813/ using document root <none> and password file <none>
I20260620 14:53:07.677742 22798 fs_manager.cc:359] Metadata directory not provided
I20260620 14:53:07.677886 22798 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 14:53:07.678241 22798 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 14:53:07.681558 22798 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskUci3DU/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781966829091300-388-0/minicluster-data/master-0/data/instance:
uuid: "9e796342088f415ba80b5bdb96cd6ff6"
format_stamp: "Formatted at 2026-06-20 14:53:07 on dist-test-slave-q9j6"
server_key: "f7bdfccac0072679d754533e469af2a2"
server_key_iv: "28593e1703a8c4d5d75081e4d2b9be2c"
server_key_version: "encryptionkey@0"
I20260620 14:53:07.682439 22798 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskUci3DU/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781966829091300-388-0/minicluster-data/master-0/wal/instance:
uuid: "9e796342088f415ba80b5bdb96cd6ff6"
format_stamp: "Formatted at 2026-06-20 14:53:07 on dist-test-slave-q9j6"
server_key: "f7bdfccac0072679d754533e469af2a2"
server_key_iv: "28593e1703a8c4d5d75081e4d2b9be2c"
server_key_version: "encryptionkey@0"
I20260620 14:53:07.688249 22798 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.007s sys 0.000s
I20260620 14:53:07.692112 22813 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 14:53:07.693606 22798 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.001s
I20260620 14:53:07.693825 22798 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskUci3DU/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781966829091300-388-0/minicluster-data/master-0/data,/tmp/dist-test-taskUci3DU/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781966829091300-388-0/minicluster-data/master-0/wal
uuid: "9e796342088f415ba80b5bdb96cd6ff6"
format_stamp: "Formatted at 2026-06-20 14:53:07 on dist-test-slave-q9j6"
server_key: "f7bdfccac0072679d754533e469af2a2"
server_key_iv: "28593e1703a8c4d5d75081e4d2b9be2c"
server_key_version: "encryptionkey@0"
I20260620 14:53:07.694090 22798 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskUci3DU/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781966829091300-388-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskUci3DU/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781966829091300-388-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskUci3DU/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781966829091300-388-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 14:53:07.715847 22798 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 14:53:07.720530 22798 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 14:53:07.720857 22798 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 14:53:07.740602 22798 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.62:45139
I20260620 14:53:07.740578 22864 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.62:45139 every 8 connection(s)
I20260620 14:53:07.742092 22798 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskUci3DU/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781966829091300-388-0/minicluster-data/master-0/data/info.pb
I20260620 14:53:07.746001 22865 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 14:53:07.751310 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskUci3DU/build/asan/bin/kudu as pid 22798
I20260620 14:53:07.751554 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskUci3DU/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781966829091300-388-0/minicluster-data/master-0/wal/instance
I20260620 14:53:07.752234 388 external_mini_cluster.cc:1428] Setting key dd97d6e0ea2d0c53fd7e79146cb0d888
I20260620 14:53:07.753334 22865 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.97.62" port: 45139 } has no permanent_uuid. Determining permanent_uuid...
Jun 20 14:53:07 dist-test-slave-q9j6 krb5kdc[22785](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781967187, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.62@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
WARNING: no policy specified for kudu/127.0.97.61@KRBTEST.COM; defaulting to no policy
I20260620 14:53:07.780936 22867 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:53:07.757216 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42376 (local address 127.0.97.62:45139)
0620 14:53:07.758239 (+ 1023us) server_negotiation.cc:206] Beginning negotiation
0620 14:53:07.758264 (+ 25us) server_negotiation.cc:399] Waiting for connection header
0620 14:53:07.758818 (+ 554us) server_negotiation.cc:407] Connection header received
0620 14:53:07.762140 (+ 3322us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:53:07.762150 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:53:07.762542 (+ 392us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:53:07.762814 (+ 272us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:53:07.764151 (+ 1337us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:53:07.765402 (+ 1251us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:53:07.766847 (+ 1445us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:53:07.767442 (+ 595us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:53:07.767697 (+ 255us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:53:07.771643 (+ 3946us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 14:53:07.771659 (+ 16us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 14:53:07.771681 (+ 22us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 14:53:07.771737 (+ 56us) server_negotiation.cc:922] calling sasl_server_start()
0620 14:53:07.776110 (+ 4373us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:53:07.776963 (+ 853us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:53:07.776974 (+ 11us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:53:07.776980 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:53:07.777215 (+ 235us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:53:07.777660 (+ 445us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:53:07.777666 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:53:07.777670 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:53:07.778477 (+ 807us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 14:53:07.778718 (+ 241us) server_negotiation.cc:1035] Waiting for connection context
0620 14:53:07.779342 (+ 624us) server_negotiation.cc:299] Negotiation successful
0620 14:53:07.779834 (+ 492us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":200,"server-negotiator.queue_time_us":723,"thread_start_us":458,"threads_started":1}
I20260620 14:53:07.780973 22868 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:53:07.757215 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:45139 (local address 127.0.0.1:42376)
0620 14:53:07.758601 (+ 1386us) negotiation.cc:107] Waiting for socket to connect
0620 14:53:07.758655 (+ 54us) client_negotiation.cc:174] Beginning negotiation
0620 14:53:07.761649 (+ 2994us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 14:53:07.762825 (+ 1176us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 14:53:07.762830 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 14:53:07.763445 (+ 615us) client_negotiation.cc:189] Negotiated authn=SASL
0620 14:53:07.763901 (+ 456us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:53:07.763923 (+ 22us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:53:07.765578 (+ 1655us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:53:07.765589 (+ 11us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:53:07.766690 (+ 1101us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:53:07.766702 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:53:07.767730 (+ 1028us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:53:07.767738 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:53:07.767907 (+ 169us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:53:07.768666 (+ 759us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 14:53:07.768690 (+ 24us) client_negotiation.cc:656] calling sasl_client_start()
0620 14:53:07.771308 (+ 2618us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 14:53:07.776328 (+ 5020us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 14:53:07.776339 (+ 11us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 14:53:07.776350 (+ 11us) client_negotiation.cc:760] calling sasl_client_step()
0620 14:53:07.776784 (+ 434us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 14:53:07.777351 (+ 567us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 14:53:07.777354 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 14:53:07.777356 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 14:53:07.777526 (+ 170us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 14:53:07.778661 (+ 1135us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 14:53:07.778668 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 14:53:07.778876 (+ 208us) client_negotiation.cc:769] Sending connection context
0620 14:53:07.779121 (+ 245us) client_negotiation.cc:240] Negotiation successful
0620 14:53:07.779693 (+ 572us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1054,"spinlock_wait_cycles":59136,"thread_start_us":821,"threads_started":1}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.61@KRBTEST.COM" created.
I20260620 14:53:07.784198 22865 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.97.61" port: 45529 } has no permanent_uuid. Determining permanent_uuid...
I20260620 14:53:07.785354 22868 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:53:07.784940 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.61:45529 (local address 127.0.0.1:41464)
0620 14:53:07.785116 (+ 176us) negotiation.cc:107] Waiting for socket to connect
0620 14:53:07.785227 (+ 111us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.61:45529: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":65}
W20260620 14:53:07.785607 22814 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.97.61:45529: connect: Connection refused (error 111)
I20260620 14:53:07.787243 22868 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:53:07.786851 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.61:45529 (local address 127.0.0.1:41476)
0620 14:53:07.787026 (+ 175us) negotiation.cc:107] Waiting for socket to connect
0620 14:53:07.787108 (+ 82us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.61:45529: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":61}
W20260620 14:53:07.787461 22865 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.97.61:45529: Network error: Client connection negotiation failed: client connection to 127.0.97.61:45529: connect: Connection refused (error 111)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.61 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskUci3DU/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781966829091300-388-0/krb5kdc/kudu_127.0.97.61.keytab.
Entry for principal kudu/127.0.97.61 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskUci3DU/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781966829091300-388-0/krb5kdc/kudu_127.0.97.61.keytab.
I20260620 14:53:07.814342 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskUci3DU/build/asan/bin/kudu
/tmp/dist-test-taskUci3DU/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskUci3DU/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781966829091300-388-0/minicluster-data/master-1/wal
--fs_data_dirs=/tmp/dist-test-taskUci3DU/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781966829091300-388-0/minicluster-data/master-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskUci3DU/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781966829091300-388-0/minicluster-data/master-1/logs
--server_dump_info_path=/tmp/dist-test-taskUci3DU/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781966829091300-388-0/minicluster-data/master-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.61:45529
--webserver_interface=127.0.97.61
--webserver_port=0
--keytab_file=/tmp/dist-test-taskUci3DU/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781966829091300-388-0/krb5kdc/kudu_127.0.97.61.keytab
--principal=kudu/127.0.97.61
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:37801
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.97.62:45139,127.0.97.61:45529,127.0.97.60:33721
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=optional with env {KRB5CCNAME=/tmp/dist-test-taskUci3DU/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781966829091300-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskUci3DU/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781966829091300-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskUci3DU/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781966829091300-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260620 14:53:07.836906 22865 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.97.61" port: 45529 } attempt: 1
I20260620 14:53:07.838340 22868 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:53:07.837709 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.61:45529 (local address 127.0.0.1:41486)
0620 14:53:07.837981 (+ 272us) negotiation.cc:107] Waiting for socket to connect
0620 14:53:07.838125 (+ 144us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.61:45529: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":97}
I20260620 14:53:07.839555 22868 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:53:07.839070 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.61:45529 (local address 127.0.0.1:41488)
0620 14:53:07.839264 (+ 194us) negotiation.cc:107] Waiting for socket to connect
0620 14:53:07.839368 (+ 104us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.61:45529: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":60}
W20260620 14:53:07.839790 22865 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.97.61:45529: Network error: Client connection negotiation failed: client connection to 127.0.97.61:45529: connect: Connection refused (error 111)
I20260620 14:53:07.908099 22865 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.97.61" port: 45529 } attempt: 2
I20260620 14:53:07.909409 22868 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:53:07.908836 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.61:45529 (local address 127.0.0.1:41494)
0620 14:53:07.909121 (+ 285us) negotiation.cc:107] Waiting for socket to connect
0620 14:53:07.909233 (+ 112us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.61:45529: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":86}
I20260620 14:53:07.910580 22868 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:53:07.910126 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.61:45529 (local address 127.0.0.1:41502)
0620 14:53:07.910299 (+ 173us) negotiation.cc:107] Waiting for socket to connect
0620 14:53:07.910406 (+ 107us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.61:45529: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":62}
W20260620 14:53:07.910816 22865 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.97.61:45529: Network error: Client connection negotiation failed: client connection to 127.0.97.61:45529: connect: Connection refused (error 111)
I20260620 14:53:08.002144 22865 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.97.61" port: 45529 } attempt: 3
I20260620 14:53:08.003522 22868 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:53:08.002961 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.61:45529 (local address 127.0.0.1:41504)
0620 14:53:08.003218 (+ 257us) negotiation.cc:107] Waiting for socket to connect
0620 14:53:08.003339 (+ 121us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.61:45529: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":97}
I20260620 14:53:08.004812 22868 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:53:08.004353 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.61:45529 (local address 127.0.0.1:41518)
0620 14:53:08.004524 (+ 171us) negotiation.cc:107] Waiting for socket to connect
0620 14:53:08.004625 (+ 101us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.61:45529: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":57}
W20260620 14:53:08.005079 22865 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.97.61:45529: Network error: Client connection negotiation failed: client connection to 127.0.97.61:45529: connect: Connection refused (error 111)
I20260620 14:53:08.148394 22865 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.97.61" port: 45529 } attempt: 4
I20260620 14:53:08.149782 22868 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:53:08.149226 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.61:45529 (local address 127.0.0.1:41524)
0620 14:53:08.149549 (+ 323us) negotiation.cc:107] Waiting for socket to connect
0620 14:53:08.149641 (+ 92us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.61:45529: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":177}
I20260620 14:53:08.150954 22868 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:53:08.150577 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.61:45529 (local address 127.0.0.1:41536)
0620 14:53:08.150722 (+ 145us) negotiation.cc:107] Waiting for socket to connect
0620 14:53:08.150815 (+ 93us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.61:45529: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":51}
W20260620 14:53:08.151194 22865 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.97.61:45529: Network error: Client connection negotiation failed: client connection to 127.0.97.61:45529: connect: Connection refused (error 111)
W20260620 14:53:08.167303 22872 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260620 14:53:08.167778 22872 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 14:53:08.167914 22872 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 14:53:08.168056 22872 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 14:53:08.175511 22872 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 14:53:08.175706 22872 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 14:53:08.175830 22872 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 14:53:08.175926 22872 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 14:53:08.176017 22872 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 14:53:08.186169 22872 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:37801
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-taskUci3DU/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781966829091300-388-0/minicluster-data/master-1/data
--fs_wal_dir=/tmp/dist-test-taskUci3DU/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781966829091300-388-0/minicluster-data/master-1/wal
--ipki_ca_key_size=768
--master_addresses=127.0.97.62:45139,127.0.97.61:45529,127.0.97.60:33721
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskUci3DU/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781966829091300-388-0/krb5kdc/kudu_127.0.97.61.keytab
--principal=kudu/127.0.97.61
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.61:45529
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskUci3DU/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781966829091300-388-0/minicluster-data/master-1/data/info.pb
--webserver_interface=127.0.97.61
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskUci3DU/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781966829091300-388-0/minicluster-data/master-1/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 13:41:20 UTC on ccycloud.quasar-vdagoi.root.comops.site
build id 4457462
ASAN enabled
I20260620 14:53:08.187795 22872 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 14:53:08.189221 22872 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 14:53:08.201675 22877 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 14:53:08.201843 22878 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 14:53:08.201961 22880 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 14:53:08.203346 22872 server_base.cc:1034] running on GCE node
I20260620 14:53:08.204288 22872 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 14:53:08.205926 22872 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 14:53:08.207202 22872 hybrid_clock.cc:648] HybridClock initialized: now 1781967188207120 us; error 73 us; skew 500 ppm
I20260620 14:53:08.207573 22872 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 14:53:08 dist-test-slave-q9j6 krb5kdc[22785](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781967188, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.61@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 14:53:08.211210 22872 init.cc:377] Logged in from keytab as kudu/127.0.97.61@KRBTEST.COM (short username kudu)
I20260620 14:53:08.213011 22872 webserver.cc:460] Webserver started at http://127.0.97.61:42003/ using document root <none> and password file <none>
I20260620 14:53:08.213797 22872 fs_manager.cc:359] Metadata directory not provided
I20260620 14:53:08.213943 22872 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 14:53:08.214308 22872 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 14:53:08.217600 22872 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskUci3DU/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781966829091300-388-0/minicluster-data/master-1/data/instance:
uuid: "3ef16e7478cd4f99b109748e7c962f1e"
format_stamp: "Formatted at 2026-06-20 14:53:08 on dist-test-slave-q9j6"
server_key: "0041b8f8236a9cfd5d798823614d6801"
server_key_iv: "d91720cfaaf390912eade328e52bedbb"
server_key_version: "encryptionkey@0"
I20260620 14:53:08.218489 22872 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskUci3DU/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781966829091300-388-0/minicluster-data/master-1/wal/instance:
uuid: "3ef16e7478cd4f99b109748e7c962f1e"
format_stamp: "Formatted at 2026-06-20 14:53:08 on dist-test-slave-q9j6"
server_key: "0041b8f8236a9cfd5d798823614d6801"
server_key_iv: "d91720cfaaf390912eade328e52bedbb"
server_key_version: "encryptionkey@0"
I20260620 14:53:08.224467 22872 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.007s sys 0.000s
I20260620 14:53:08.228461 22887 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 14:53:08.230010 22872 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260620 14:53:08.230247 22872 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskUci3DU/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781966829091300-388-0/minicluster-data/master-1/data,/tmp/dist-test-taskUci3DU/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781966829091300-388-0/minicluster-data/master-1/wal
uuid: "3ef16e7478cd4f99b109748e7c962f1e"
format_stamp: "Formatted at 2026-06-20 14:53:08 on dist-test-slave-q9j6"
server_key: "0041b8f8236a9cfd5d798823614d6801"
server_key_iv: "d91720cfaaf390912eade328e52bedbb"
server_key_version: "encryptionkey@0"
I20260620 14:53:08.230495 22872 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskUci3DU/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781966829091300-388-0/minicluster-data/master-1/wal
metadata directory: /tmp/dist-test-taskUci3DU/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781966829091300-388-0/minicluster-data/master-1/wal
1 data directories: /tmp/dist-test-taskUci3DU/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781966829091300-388-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 14:53:08.257369 22872 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 14:53:08.262401 22872 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 14:53:08.262724 22872 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 14:53:08.283982 22938 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.61:45529 every 8 connection(s)
I20260620 14:53:08.283977 22872 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.61:45529
I20260620 14:53:08.285619 22872 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskUci3DU/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781966829091300-388-0/minicluster-data/master-1/data/info.pb
I20260620 14:53:08.289841 22939 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 14:53:08.291764 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskUci3DU/build/asan/bin/kudu as pid 22872
I20260620 14:53:08.291950 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskUci3DU/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781966829091300-388-0/minicluster-data/master-1/wal/instance
I20260620 14:53:08.292390 388 external_mini_cluster.cc:1428] Setting key 2a6b92d20940b6d77753a2094b67422b
I20260620 14:53:08.297180 22939 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.97.62" port: 45139 } has no permanent_uuid. Determining permanent_uuid...
Jun 20 14:53:08 dist-test-slave-q9j6 krb5kdc[22785](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781967188, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.61@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
WARNING: no policy specified for kudu/127.0.97.60@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.60@KRBTEST.COM" created.
I20260620 14:53:08.323966 22942 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:53:08.300374 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42386 (local address 127.0.97.62:45139)
0620 14:53:08.300984 (+ 610us) server_negotiation.cc:206] Beginning negotiation
0620 14:53:08.300991 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0620 14:53:08.302107 (+ 1116us) server_negotiation.cc:407] Connection header received
0620 14:53:08.305197 (+ 3090us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:53:08.305202 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:53:08.305280 (+ 78us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:53:08.305382 (+ 102us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:53:08.307822 (+ 2440us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:53:08.308630 (+ 808us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:53:08.311317 (+ 2687us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:53:08.311872 (+ 555us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:53:08.312002 (+ 130us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:53:08.317076 (+ 5074us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 14:53:08.317083 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 14:53:08.317086 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 14:53:08.317131 (+ 45us) server_negotiation.cc:922] calling sasl_server_start()
0620 14:53:08.320308 (+ 3177us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:53:08.321235 (+ 927us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:53:08.321241 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:53:08.321243 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:53:08.321375 (+ 132us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:53:08.321927 (+ 552us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:53:08.321939 (+ 12us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:53:08.321942 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:53:08.322242 (+ 300us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 14:53:08.322419 (+ 177us) server_negotiation.cc:1035] Waiting for connection context
0620 14:53:08.323247 (+ 828us) server_negotiation.cc:299] Negotiation successful
0620 14:53:08.323515 (+ 268us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":456,"thread_start_us":302,"threads_started":1}
I20260620 14:53:08.325034 22943 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:53:08.301024 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:45139 (local address 127.0.0.1:42386)
0620 14:53:08.301901 (+ 877us) negotiation.cc:107] Waiting for socket to connect
0620 14:53:08.301953 (+ 52us) client_negotiation.cc:174] Beginning negotiation
0620 14:53:08.304855 (+ 2902us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 14:53:08.305699 (+ 844us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 14:53:08.305714 (+ 15us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 14:53:08.306595 (+ 881us) client_negotiation.cc:189] Negotiated authn=SASL
0620 14:53:08.307474 (+ 879us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:53:08.307505 (+ 31us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:53:08.308832 (+ 1327us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:53:08.308840 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:53:08.311094 (+ 2254us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:53:08.311116 (+ 22us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:53:08.312159 (+ 1043us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:53:08.312193 (+ 34us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:53:08.312569 (+ 376us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:53:08.313446 (+ 877us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 14:53:08.313484 (+ 38us) client_negotiation.cc:656] calling sasl_client_start()
0620 14:53:08.316741 (+ 3257us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 14:53:08.320514 (+ 3773us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 14:53:08.320526 (+ 12us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 14:53:08.320554 (+ 28us) client_negotiation.cc:760] calling sasl_client_step()
0620 14:53:08.321067 (+ 513us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 14:53:08.321509 (+ 442us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 14:53:08.321514 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 14:53:08.321518 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0620 14:53:08.321796 (+ 278us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 14:53:08.322385 (+ 589us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 14:53:08.322394 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 14:53:08.322835 (+ 441us) client_negotiation.cc:769] Sending connection context
0620 14:53:08.323148 (+ 313us) client_negotiation.cc:240] Negotiation successful
0620 14:53:08.323772 (+ 624us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":535,"thread_start_us":315,"threads_started":1}
I20260620 14:53:08.327060 22939 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.97.61" port: 45529 } has no permanent_uuid. Determining permanent_uuid...
Jun 20 14:53:08 dist-test-slave-q9j6 krb5kdc[22785](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781967188, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.61@KRBTEST.COM for kudu/127.0.97.61@KRBTEST.COM
I20260620 14:53:08.344250 22944 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:53:08.328361 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41550 (local address 127.0.97.61:45529)
0620 14:53:08.329029 (+ 668us) server_negotiation.cc:206] Beginning negotiation
0620 14:53:08.329035 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 14:53:08.329062 (+ 27us) server_negotiation.cc:407] Connection header received
0620 14:53:08.329112 (+ 50us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:53:08.329118 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:53:08.329320 (+ 202us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:53:08.329509 (+ 189us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:53:08.330287 (+ 778us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:53:08.331378 (+ 1091us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:53:08.332630 (+ 1252us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:53:08.333201 (+ 571us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:53:08.333352 (+ 151us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:53:08.337397 (+ 4045us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 14:53:08.337414 (+ 17us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 14:53:08.337432 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 14:53:08.337471 (+ 39us) server_negotiation.cc:922] calling sasl_server_start()
0620 14:53:08.340744 (+ 3273us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:53:08.341745 (+ 1001us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:53:08.341754 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:53:08.341756 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:53:08.341891 (+ 135us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:53:08.342372 (+ 481us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:53:08.342375 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:53:08.342378 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:53:08.342763 (+ 385us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 14:53:08.343033 (+ 270us) server_negotiation.cc:1035] Waiting for connection context
0620 14:53:08.343513 (+ 480us) server_negotiation.cc:299] Negotiation successful
0620 14:53:08.343927 (+ 414us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":88,"server-negotiator.queue_time_us":521,"thread_start_us":389,"threads_started":1}
I20260620 14:53:08.344255 22943 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:53:08.328041 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.61:45529 (local address 127.0.0.1:41550)
0620 14:53:08.328228 (+ 187us) negotiation.cc:107] Waiting for socket to connect
0620 14:53:08.328245 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0620 14:53:08.328342 (+ 97us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 14:53:08.329485 (+ 1143us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 14:53:08.329489 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 14:53:08.329965 (+ 476us) client_negotiation.cc:189] Negotiated authn=SASL
0620 14:53:08.330103 (+ 138us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:53:08.330112 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:53:08.331555 (+ 1443us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:53:08.331561 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:53:08.332475 (+ 914us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:53:08.332488 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:53:08.333373 (+ 885us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:53:08.333379 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:53:08.333551 (+ 172us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:53:08.334262 (+ 711us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 14:53:08.334280 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0620 14:53:08.337093 (+ 2813us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 14:53:08.340943 (+ 3850us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 14:53:08.340952 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 14:53:08.340959 (+ 7us) client_negotiation.cc:760] calling sasl_client_step()
0620 14:53:08.341567 (+ 608us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 14:53:08.342032 (+ 465us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 14:53:08.342040 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 14:53:08.342047 (+ 7us) client_negotiation.cc:760] calling sasl_client_step()
0620 14:53:08.342256 (+ 209us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 14:53:08.342936 (+ 680us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 14:53:08.342945 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 14:53:08.343162 (+ 217us) client_negotiation.cc:769] Sending connection context
0620 14:53:08.343339 (+ 177us) client_negotiation.cc:240] Negotiation successful
0620 14:53:08.343748 (+ 409us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":60}
I20260620 14:53:08.347385 22939 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.97.60" port: 33721 } has no permanent_uuid. Determining permanent_uuid...
I20260620 14:53:08.348678 22943 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:53:08.348156 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.60:33721 (local address 127.0.0.1:36856)
0620 14:53:08.348347 (+ 191us) negotiation.cc:107] Waiting for socket to connect
0620 14:53:08.348526 (+ 179us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.60:33721: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":73}
W20260620 14:53:08.348973 22890 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.97.60:33721: connect: Connection refused (error 111)
I20260620 14:53:08.350660 22943 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:53:08.350306 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.60:33721 (local address 127.0.0.1:36858)
0620 14:53:08.350460 (+ 154us) negotiation.cc:107] Waiting for socket to connect
0620 14:53:08.350546 (+ 86us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.60:33721: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":54}
W20260620 14:53:08.350927 22939 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.97.60:33721: Network error: Client connection negotiation failed: client connection to 127.0.97.60:33721: connect: Connection refused (error 111)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.60 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskUci3DU/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781966829091300-388-0/krb5kdc/kudu_127.0.97.60.keytab.
Entry for principal kudu/127.0.97.60 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskUci3DU/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781966829091300-388-0/krb5kdc/kudu_127.0.97.60.keytab.
I20260620 14:53:08.356843 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskUci3DU/build/asan/bin/kudu
/tmp/dist-test-taskUci3DU/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskUci3DU/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781966829091300-388-0/minicluster-data/master-2/wal
--fs_data_dirs=/tmp/dist-test-taskUci3DU/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781966829091300-388-0/minicluster-data/master-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskUci3DU/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781966829091300-388-0/minicluster-data/master-2/logs
--server_dump_info_path=/tmp/dist-test-taskUci3DU/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781966829091300-388-0/minicluster-data/master-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.60:33721
--webserver_interface=127.0.97.60
--webserver_port=0
--keytab_file=/tmp/dist-test-taskUci3DU/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781966829091300-388-0/krb5kdc/kudu_127.0.97.60.keytab
--principal=kudu/127.0.97.60
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:37801
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.97.62:45139,127.0.97.61:45529,127.0.97.60:33721
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=optional with env {KRB5CCNAME=/tmp/dist-test-taskUci3DU/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781966829091300-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskUci3DU/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781966829091300-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskUci3DU/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781966829091300-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260620 14:53:08.400213 22939 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.97.60" port: 33721 } attempt: 1
I20260620 14:53:08.401459 22943 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:53:08.400993 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.60:33721 (local address 127.0.0.1:36868)
0620 14:53:08.401201 (+ 208us) negotiation.cc:107] Waiting for socket to connect
0620 14:53:08.401296 (+ 95us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.60:33721: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":87}
I20260620 14:53:08.402545 22943 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:53:08.402165 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.60:33721 (local address 127.0.0.1:36876)
0620 14:53:08.402330 (+ 165us) negotiation.cc:107] Waiting for socket to connect
0620 14:53:08.402429 (+ 99us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.60:33721: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":55}
W20260620 14:53:08.402773 22939 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.97.60:33721: Network error: Client connection negotiation failed: client connection to 127.0.97.60:33721: connect: Connection refused (error 111)
I20260620 14:53:08.450469 22865 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.97.61" port: 45529 } attempt: 5
Jun 20 14:53:08 dist-test-slave-q9j6 krb5kdc[22785](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781967187, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.62@KRBTEST.COM for kudu/127.0.97.61@KRBTEST.COM
I20260620 14:53:08.466090 22868 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:53:08.451218 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.61:45529 (local address 127.0.0.1:41552)
0620 14:53:08.451444 (+ 226us) negotiation.cc:107] Waiting for socket to connect
0620 14:53:08.451468 (+ 24us) client_negotiation.cc:174] Beginning negotiation
0620 14:53:08.451643 (+ 175us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 14:53:08.452054 (+ 411us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 14:53:08.452059 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 14:53:08.452581 (+ 522us) client_negotiation.cc:189] Negotiated authn=SASL
0620 14:53:08.452825 (+ 244us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:53:08.452837 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:53:08.453880 (+ 1043us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:53:08.453887 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:53:08.455243 (+ 1356us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:53:08.455260 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:53:08.456100 (+ 840us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:53:08.456107 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:53:08.456300 (+ 193us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:53:08.456990 (+ 690us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 14:53:08.457022 (+ 32us) client_negotiation.cc:656] calling sasl_client_start()
0620 14:53:08.459582 (+ 2560us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 14:53:08.463494 (+ 3912us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 14:53:08.463499 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 14:53:08.463502 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0620 14:53:08.464020 (+ 518us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 14:53:08.464474 (+ 454us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 14:53:08.464477 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 14:53:08.464479 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 14:53:08.464617 (+ 138us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 14:53:08.465302 (+ 685us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 14:53:08.465306 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 14:53:08.465447 (+ 141us) client_negotiation.cc:769] Sending connection context
0620 14:53:08.465555 (+ 108us) client_negotiation.cc:240] Negotiation successful
0620 14:53:08.465805 (+ 250us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":66}
I20260620 14:53:08.466480 22944 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:53:08.451360 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41552 (local address 127.0.97.61:45529)
0620 14:53:08.451550 (+ 190us) server_negotiation.cc:206] Beginning negotiation
0620 14:53:08.451555 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 14:53:08.451575 (+ 20us) server_negotiation.cc:407] Connection header received
0620 14:53:08.451801 (+ 226us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:53:08.451805 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:53:08.451871 (+ 66us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:53:08.451966 (+ 95us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:53:08.452990 (+ 1024us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:53:08.453696 (+ 706us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:53:08.455418 (+ 1722us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:53:08.455933 (+ 515us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:53:08.456031 (+ 98us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:53:08.459911 (+ 3880us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 14:53:08.459921 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 14:53:08.459928 (+ 7us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 14:53:08.459984 (+ 56us) server_negotiation.cc:922] calling sasl_server_start()
0620 14:53:08.463312 (+ 3328us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:53:08.464169 (+ 857us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:53:08.464178 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:53:08.464183 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:53:08.464345 (+ 162us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:53:08.464741 (+ 396us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:53:08.464747 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:53:08.464750 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:53:08.465172 (+ 422us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 14:53:08.465359 (+ 187us) server_negotiation.cc:1035] Waiting for connection context
0620 14:53:08.465751 (+ 392us) server_negotiation.cc:299] Negotiation successful
0620 14:53:08.466073 (+ 322us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":67}
I20260620 14:53:08.467662 22865 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.97.60" port: 33721 } has no permanent_uuid. Determining permanent_uuid...
I20260620 14:53:08.468806 22868 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:53:08.468314 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.60:33721 (local address 127.0.0.1:36890)
0620 14:53:08.468512 (+ 198us) negotiation.cc:107] Waiting for socket to connect
0620 14:53:08.468614 (+ 102us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.60:33721: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":66}
I20260620 14:53:08.469833 22868 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:53:08.469434 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.60:33721 (local address 127.0.0.1:36902)
0620 14:53:08.469601 (+ 167us) negotiation.cc:107] Waiting for socket to connect
0620 14:53:08.469705 (+ 104us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.60:33721: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":58}
W20260620 14:53:08.470058 22865 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.97.60:33721: Network error: Client connection negotiation failed: client connection to 127.0.97.60:33721: connect: Connection refused (error 111)
I20260620 14:53:08.470990 22939 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.97.60" port: 33721 } attempt: 2
I20260620 14:53:08.472005 22943 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:53:08.471562 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.60:33721 (local address 127.0.0.1:36904)
0620 14:53:08.471771 (+ 209us) negotiation.cc:107] Waiting for socket to connect
0620 14:53:08.471864 (+ 93us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.60:33721: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":88}
I20260620 14:53:08.472990 22943 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:53:08.472657 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.60:33721 (local address 127.0.0.1:36914)
0620 14:53:08.472805 (+ 148us) negotiation.cc:107] Waiting for socket to connect
0620 14:53:08.472885 (+ 80us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.60:33721: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":54}
W20260620 14:53:08.473219 22939 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.97.60:33721: Network error: Client connection negotiation failed: client connection to 127.0.97.60:33721: connect: Connection refused (error 111)
I20260620 14:53:08.521319 22865 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.97.60" port: 33721 } attempt: 1
I20260620 14:53:08.522780 22868 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:53:08.522088 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.60:33721 (local address 127.0.0.1:36916)
0620 14:53:08.522377 (+ 289us) negotiation.cc:107] Waiting for socket to connect
0620 14:53:08.522519 (+ 142us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.60:33721: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":98}
I20260620 14:53:08.524072 22868 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:53:08.523584 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.60:33721 (local address 127.0.0.1:36924)
0620 14:53:08.523804 (+ 220us) negotiation.cc:107] Waiting for socket to connect
0620 14:53:08.523907 (+ 103us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.60:33721: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":105}
W20260620 14:53:08.524343 22865 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.97.60:33721: Network error: Client connection negotiation failed: client connection to 127.0.97.60:33721: connect: Connection refused (error 111)
I20260620 14:53:08.564481 22939 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.97.60" port: 33721 } attempt: 3
I20260620 14:53:08.565795 22943 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:53:08.565206 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.60:33721 (local address 127.0.0.1:36930)
0620 14:53:08.565471 (+ 265us) negotiation.cc:107] Waiting for socket to connect
0620 14:53:08.565592 (+ 121us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.60:33721: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":86}
I20260620 14:53:08.567130 22943 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:53:08.566604 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.60:33721 (local address 127.0.0.1:36944)
0620 14:53:08.566854 (+ 250us) negotiation.cc:107] Waiting for socket to connect
0620 14:53:08.566958 (+ 104us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.60:33721: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":140}
W20260620 14:53:08.567390 22939 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.97.60:33721: Network error: Client connection negotiation failed: client connection to 127.0.97.60:33721: connect: Connection refused (error 111)
I20260620 14:53:08.592595 22865 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.97.60" port: 33721 } attempt: 2
I20260620 14:53:08.593940 22868 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:53:08.593349 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.60:33721 (local address 127.0.0.1:36948)
0620 14:53:08.593610 (+ 261us) negotiation.cc:107] Waiting for socket to connect
0620 14:53:08.593732 (+ 122us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.60:33721: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":94}
I20260620 14:53:08.595091 22868 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:53:08.594675 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.60:33721 (local address 127.0.0.1:36954)
0620 14:53:08.594855 (+ 180us) negotiation.cc:107] Waiting for socket to connect
0620 14:53:08.594946 (+ 91us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.60:33721: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":52}
W20260620 14:53:08.595330 22865 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.97.60:33721: Network error: Client connection negotiation failed: client connection to 127.0.97.60:33721: connect: Connection refused (error 111)
I20260620 14:53:08.701575 22865 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.97.60" port: 33721 } attempt: 3
I20260620 14:53:08.702826 22868 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:53:08.702276 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.60:33721 (local address 127.0.0.1:36958)
0620 14:53:08.702510 (+ 234us) negotiation.cc:107] Waiting for socket to connect
0620 14:53:08.702617 (+ 107us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.60:33721: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":83}
I20260620 14:53:08.704149 22868 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:53:08.703643 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.60:33721 (local address 127.0.0.1:36970)
0620 14:53:08.703884 (+ 241us) negotiation.cc:107] Waiting for socket to connect
0620 14:53:08.703983 (+ 99us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.60:33721: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":104}
W20260620 14:53:08.704418 22865 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.97.60:33721: Network error: Client connection negotiation failed: client connection to 127.0.97.60:33721: connect: Connection refused (error 111)
I20260620 14:53:08.710671 22939 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.97.60" port: 33721 } attempt: 4
I20260620 14:53:08.711997 22943 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:53:08.711394 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.60:33721 (local address 127.0.0.1:36972)
0620 14:53:08.711690 (+ 296us) negotiation.cc:107] Waiting for socket to connect
0620 14:53:08.711800 (+ 110us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.60:33721: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":77}
I20260620 14:53:08.713150 22943 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:53:08.712751 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.60:33721 (local address 127.0.0.1:36984)
0620 14:53:08.712912 (+ 161us) negotiation.cc:107] Waiting for socket to connect
0620 14:53:08.713003 (+ 91us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.60:33721: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":52}
W20260620 14:53:08.713436 22939 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.97.60:33721: Network error: Client connection negotiation failed: client connection to 127.0.97.60:33721: connect: Connection refused (error 111)
W20260620 14:53:08.726625 22947 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260620 14:53:08.727080 22947 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 14:53:08.727201 22947 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 14:53:08.727330 22947 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 14:53:08.735144 22947 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 14:53:08.735250 22947 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 14:53:08.735337 22947 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 14:53:08.735435 22947 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 14:53:08.735544 22947 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 14:53:08.746918 22947 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:37801
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-taskUci3DU/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781966829091300-388-0/minicluster-data/master-2/data
--fs_wal_dir=/tmp/dist-test-taskUci3DU/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781966829091300-388-0/minicluster-data/master-2/wal
--ipki_ca_key_size=768
--master_addresses=127.0.97.62:45139,127.0.97.61:45529,127.0.97.60:33721
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskUci3DU/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781966829091300-388-0/krb5kdc/kudu_127.0.97.60.keytab
--principal=kudu/127.0.97.60
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.60:33721
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskUci3DU/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781966829091300-388-0/minicluster-data/master-2/data/info.pb
--webserver_interface=127.0.97.60
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskUci3DU/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781966829091300-388-0/minicluster-data/master-2/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 13:41:20 UTC on ccycloud.quasar-vdagoi.root.comops.site
build id 4457462
ASAN enabled
I20260620 14:53:08.748409 22947 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 14:53:08.749815 22947 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 14:53:08.762285 22953 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 14:53:08.762252 22952 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 14:53:08.762321 22955 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 14:53:08.763212 22947 server_base.cc:1034] running on GCE node
I20260620 14:53:08.764135 22947 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 14:53:08.765633 22947 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 14:53:08.766865 22947 hybrid_clock.cc:648] HybridClock initialized: now 1781967188766799 us; error 39 us; skew 500 ppm
I20260620 14:53:08.767189 22947 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 14:53:08 dist-test-slave-q9j6 krb5kdc[22785](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781967188, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.60@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 14:53:08.773450 22947 init.cc:377] Logged in from keytab as kudu/127.0.97.60@KRBTEST.COM (short username kudu)
I20260620 14:53:08.775164 22947 webserver.cc:460] Webserver started at http://127.0.97.60:40047/ using document root <none> and password file <none>
I20260620 14:53:08.776049 22947 fs_manager.cc:359] Metadata directory not provided
I20260620 14:53:08.776181 22947 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 14:53:08.776546 22947 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 14:53:08.779848 22947 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskUci3DU/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781966829091300-388-0/minicluster-data/master-2/data/instance:
uuid: "c261bc77236444b2a4df214340c6afde"
format_stamp: "Formatted at 2026-06-20 14:53:08 on dist-test-slave-q9j6"
server_key: "43140d229cd00a493a60a1f5021974de"
server_key_iv: "9585931b19cf173f64ff98046e93f52a"
server_key_version: "encryptionkey@0"
I20260620 14:53:08.780764 22947 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskUci3DU/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781966829091300-388-0/minicluster-data/master-2/wal/instance:
uuid: "c261bc77236444b2a4df214340c6afde"
format_stamp: "Formatted at 2026-06-20 14:53:08 on dist-test-slave-q9j6"
server_key: "43140d229cd00a493a60a1f5021974de"
server_key_iv: "9585931b19cf173f64ff98046e93f52a"
server_key_version: "encryptionkey@0"
I20260620 14:53:08.786497 22947 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.000s
I20260620 14:53:08.790679 22962 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 14:53:08.792291 22947 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260620 14:53:08.792526 22947 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskUci3DU/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781966829091300-388-0/minicluster-data/master-2/data,/tmp/dist-test-taskUci3DU/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781966829091300-388-0/minicluster-data/master-2/wal
uuid: "c261bc77236444b2a4df214340c6afde"
format_stamp: "Formatted at 2026-06-20 14:53:08 on dist-test-slave-q9j6"
server_key: "43140d229cd00a493a60a1f5021974de"
server_key_iv: "9585931b19cf173f64ff98046e93f52a"
server_key_version: "encryptionkey@0"
I20260620 14:53:08.792790 22947 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskUci3DU/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781966829091300-388-0/minicluster-data/master-2/wal
metadata directory: /tmp/dist-test-taskUci3DU/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781966829091300-388-0/minicluster-data/master-2/wal
1 data directories: /tmp/dist-test-taskUci3DU/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781966829091300-388-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 14:53:08.814378 22947 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 14:53:08.818856 22947 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 14:53:08.819097 22947 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 14:53:08.839861 22947 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.60:33721
I20260620 14:53:08.840042 23013 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.60:33721 every 8 connection(s)
I20260620 14:53:08.841398 22947 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskUci3DU/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781966829091300-388-0/minicluster-data/master-2/data/info.pb
I20260620 14:53:08.844396 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskUci3DU/build/asan/bin/kudu as pid 22947
I20260620 14:53:08.844556 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskUci3DU/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781966829091300-388-0/minicluster-data/master-2/wal/instance
I20260620 14:53:08.845026 388 external_mini_cluster.cc:1428] Setting key 693e2708b6fa2063104a8bdf28335ef4
I20260620 14:53:08.846206 23014 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 14:53:08.857705 23014 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.97.62" port: 45139 } has no permanent_uuid. Determining permanent_uuid...
Jun 20 14:53:08 dist-test-slave-q9j6 krb5kdc[22785](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781967187, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.60@KRBTEST.COM
Jun 20 14:53:08 dist-test-slave-q9j6 krb5kdc[22785](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781967188, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.60@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260620 14:53:08.872429 23017 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:53:08.847157 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36990 (local address 127.0.97.60:33721)
0620 14:53:08.848124 (+ 967us) server_negotiation.cc:206] Beginning negotiation
0620 14:53:08.848142 (+ 18us) server_negotiation.cc:399] Waiting for connection header
0620 14:53:08.848200 (+ 58us) server_negotiation.cc:407] Connection header received
0620 14:53:08.850292 (+ 2092us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:53:08.850330 (+ 38us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:53:08.851005 (+ 675us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:53:08.851616 (+ 611us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:53:08.853066 (+ 1450us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:53:08.855437 (+ 2371us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:53:08.857049 (+ 1612us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:53:08.858275 (+ 1226us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:53:08.858527 (+ 252us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:53:08.863654 (+ 5127us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 14:53:08.863668 (+ 14us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 14:53:08.863686 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 14:53:08.863750 (+ 64us) server_negotiation.cc:922] calling sasl_server_start()
0620 14:53:08.867510 (+ 3760us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:53:08.868519 (+ 1009us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:53:08.868529 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:53:08.868539 (+ 10us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:53:08.868786 (+ 247us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:53:08.869202 (+ 416us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:53:08.869208 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:53:08.869211 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:53:08.869826 (+ 615us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 14:53:08.870218 (+ 392us) server_negotiation.cc:1035] Waiting for connection context
0620 14:53:08.870694 (+ 476us) server_negotiation.cc:299] Negotiation successful
0620 14:53:08.871147 (+ 453us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":692,"thread_start_us":437,"threads_started":1}
I20260620 14:53:08.879199 23019 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:53:08.861259 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:45139 (local address 127.0.0.1:42400)
0620 14:53:08.862336 (+ 1077us) negotiation.cc:107] Waiting for socket to connect
0620 14:53:08.862393 (+ 57us) client_negotiation.cc:174] Beginning negotiation
0620 14:53:08.862599 (+ 206us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 14:53:08.863118 (+ 519us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 14:53:08.863127 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 14:53:08.864398 (+ 1271us) client_negotiation.cc:189] Negotiated authn=SASL
0620 14:53:08.864854 (+ 456us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:53:08.864869 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:53:08.865995 (+ 1126us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:53:08.866002 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:53:08.867302 (+ 1300us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:53:08.867318 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:53:08.868508 (+ 1190us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:53:08.868517 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:53:08.868751 (+ 234us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:53:08.869556 (+ 805us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 14:53:08.869582 (+ 26us) client_negotiation.cc:656] calling sasl_client_start()
0620 14:53:08.872260 (+ 2678us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 14:53:08.876426 (+ 4166us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 14:53:08.876438 (+ 12us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 14:53:08.876444 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0620 14:53:08.877028 (+ 584us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 14:53:08.877435 (+ 407us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 14:53:08.877441 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 14:53:08.877445 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0620 14:53:08.877630 (+ 185us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 14:53:08.878196 (+ 566us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 14:53:08.878207 (+ 11us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 14:53:08.878410 (+ 203us) client_negotiation.cc:769] Sending connection context
0620 14:53:08.878629 (+ 219us) client_negotiation.cc:240] Negotiation successful
0620 14:53:08.878888 (+ 259us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":721,"thread_start_us":547,"threads_started":1}
I20260620 14:53:08.879558 23018 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:53:08.861216 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42400 (local address 127.0.97.62:45139)
0620 14:53:08.861846 (+ 630us) server_negotiation.cc:206] Beginning negotiation
0620 14:53:08.861855 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0620 14:53:08.862579 (+ 724us) server_negotiation.cc:407] Connection header received
0620 14:53:08.862803 (+ 224us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:53:08.862808 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:53:08.862882 (+ 74us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:53:08.863024 (+ 142us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:53:08.865057 (+ 2033us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:53:08.865800 (+ 743us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:53:08.867504 (+ 1704us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:53:08.868218 (+ 714us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:53:08.868373 (+ 155us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:53:08.872520 (+ 4147us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 14:53:08.872526 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 14:53:08.872531 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 14:53:08.872608 (+ 77us) server_negotiation.cc:922] calling sasl_server_start()
0620 14:53:08.876242 (+ 3634us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:53:08.877158 (+ 916us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:53:08.877162 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:53:08.877164 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:53:08.877302 (+ 138us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:53:08.877747 (+ 445us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:53:08.877750 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:53:08.877752 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:53:08.878037 (+ 285us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 14:53:08.878159 (+ 122us) server_negotiation.cc:1035] Waiting for connection context
0620 14:53:08.878833 (+ 674us) server_negotiation.cc:299] Negotiation successful
0620 14:53:08.879087 (+ 254us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":459,"thread_start_us":337,"threads_started":1}
I20260620 14:53:08.881206 23014 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.97.61" port: 45529 } has no permanent_uuid. Determining permanent_uuid...
I20260620 14:53:08.881659 22865 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.97.60" port: 33721 } attempt: 4
Jun 20 14:53:08 dist-test-slave-q9j6 krb5kdc[22785](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781967187, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
Jun 20 14:53:08 dist-test-slave-q9j6 krb5kdc[22785](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781967187, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.62@KRBTEST.COM for kudu/127.0.97.60@KRBTEST.COM
Jun 20 14:53:08 dist-test-slave-q9j6 krb5kdc[22785](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781967188, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.60@KRBTEST.COM for kudu/127.0.97.61@KRBTEST.COM
I20260620 14:53:08.896181 23020 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:53:08.875745 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42402 (local address 127.0.97.62:45139)
0620 14:53:08.876506 (+ 761us) server_negotiation.cc:206] Beginning negotiation
0620 14:53:08.876511 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 14:53:08.876529 (+ 18us) server_negotiation.cc:407] Connection header received
0620 14:53:08.876574 (+ 45us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:53:08.876578 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:53:08.876641 (+ 63us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:53:08.876729 (+ 88us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:53:08.878318 (+ 1589us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:53:08.879747 (+ 1429us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:53:08.881514 (+ 1767us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:53:08.882276 (+ 762us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:53:08.882406 (+ 130us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:53:08.888647 (+ 6241us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 14:53:08.888653 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 14:53:08.888657 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 14:53:08.888731 (+ 74us) server_negotiation.cc:922] calling sasl_server_start()
0620 14:53:08.892393 (+ 3662us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:53:08.893386 (+ 993us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:53:08.893394 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:53:08.893399 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:53:08.893568 (+ 169us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:53:08.894078 (+ 510us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:53:08.894087 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:53:08.894096 (+ 9us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:53:08.894528 (+ 432us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 14:53:08.894793 (+ 265us) server_negotiation.cc:1035] Waiting for connection context
0620 14:53:08.895359 (+ 566us) server_negotiation.cc:299] Negotiation successful
0620 14:53:08.895694 (+ 335us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":594,"thread_start_us":300,"threads_started":1}
I20260620 14:53:08.899473 23019 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:53:08.882116 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.61:45529 (local address 127.0.0.1:41568)
0620 14:53:08.882450 (+ 334us) negotiation.cc:107] Waiting for socket to connect
0620 14:53:08.882474 (+ 24us) client_negotiation.cc:174] Beginning negotiation
0620 14:53:08.882598 (+ 124us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 14:53:08.883041 (+ 443us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 14:53:08.883049 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 14:53:08.883635 (+ 586us) client_negotiation.cc:189] Negotiated authn=SASL
0620 14:53:08.883830 (+ 195us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:53:08.883842 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:53:08.885513 (+ 1671us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:53:08.885519 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:53:08.887020 (+ 1501us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:53:08.887040 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:53:08.888243 (+ 1203us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:53:08.888250 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:53:08.888457 (+ 207us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:53:08.889101 (+ 644us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 14:53:08.889126 (+ 25us) client_negotiation.cc:656] calling sasl_client_start()
0620 14:53:08.892219 (+ 3093us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 14:53:08.896372 (+ 4153us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 14:53:08.896380 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 14:53:08.896383 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0620 14:53:08.896896 (+ 513us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 14:53:08.897412 (+ 516us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 14:53:08.897418 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 14:53:08.897423 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0620 14:53:08.897629 (+ 206us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 14:53:08.898442 (+ 813us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 14:53:08.898447 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 14:53:08.898610 (+ 163us) client_negotiation.cc:769] Sending connection context
0620 14:53:08.898763 (+ 153us) client_negotiation.cc:240] Negotiation successful
0620 14:53:08.899083 (+ 320us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":194,"mutex_wait_us":48}
I20260620 14:53:08.899503 22868 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:53:08.882320 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.60:33721 (local address 127.0.0.1:36996)
0620 14:53:08.882600 (+ 280us) negotiation.cc:107] Waiting for socket to connect
0620 14:53:08.882618 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0620 14:53:08.882710 (+ 92us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 14:53:08.883266 (+ 556us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 14:53:08.883274 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 14:53:08.883740 (+ 466us) client_negotiation.cc:189] Negotiated authn=SASL
0620 14:53:08.883950 (+ 210us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:53:08.883965 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:53:08.884891 (+ 926us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:53:08.884897 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:53:08.886163 (+ 1266us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:53:08.886181 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:53:08.887309 (+ 1128us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:53:08.887315 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:53:08.887506 (+ 191us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:53:08.888239 (+ 733us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 14:53:08.888263 (+ 24us) client_negotiation.cc:656] calling sasl_client_start()
0620 14:53:08.891759 (+ 3496us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 14:53:08.895962 (+ 4203us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 14:53:08.895970 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 14:53:08.895975 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0620 14:53:08.896978 (+ 1003us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 14:53:08.897636 (+ 658us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 14:53:08.897641 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 14:53:08.897644 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0620 14:53:08.897821 (+ 177us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 14:53:08.898469 (+ 648us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 14:53:08.898476 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 14:53:08.898686 (+ 210us) client_negotiation.cc:769] Sending connection context
0620 14:53:08.898816 (+ 130us) client_negotiation.cc:240] Negotiation successful
0620 14:53:08.899108 (+ 292us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":170}
I20260620 14:53:08.900115 22944 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:53:08.882254 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41568 (local address 127.0.97.61:45529)
0620 14:53:08.882461 (+ 207us) server_negotiation.cc:206] Beginning negotiation
0620 14:53:08.882468 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0620 14:53:08.882576 (+ 108us) server_negotiation.cc:407] Connection header received
0620 14:53:08.882763 (+ 187us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:53:08.882769 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:53:08.882861 (+ 92us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:53:08.882977 (+ 116us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:53:08.884021 (+ 1044us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:53:08.885304 (+ 1283us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:53:08.887207 (+ 1903us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:53:08.888051 (+ 844us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:53:08.888214 (+ 163us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:53:08.892472 (+ 4258us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 14:53:08.892482 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 14:53:08.892490 (+ 8us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 14:53:08.892538 (+ 48us) server_negotiation.cc:922] calling sasl_server_start()
0620 14:53:08.896192 (+ 3654us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:53:08.897056 (+ 864us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:53:08.897066 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:53:08.897069 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:53:08.897271 (+ 202us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:53:08.897772 (+ 501us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:53:08.897778 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:53:08.897806 (+ 28us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:53:08.898207 (+ 401us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 14:53:08.898380 (+ 173us) server_negotiation.cc:1035] Waiting for connection context
0620 14:53:08.898931 (+ 551us) server_negotiation.cc:299] Negotiation successful
0620 14:53:08.899311 (+ 380us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":76}
I20260620 14:53:08.901259 23017 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:53:08.882723 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36996 (local address 127.0.97.60:33721)
0620 14:53:08.882894 (+ 171us) server_negotiation.cc:206] Beginning negotiation
0620 14:53:08.882903 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0620 14:53:08.882922 (+ 19us) server_negotiation.cc:407] Connection header received
0620 14:53:08.882972 (+ 50us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:53:08.882976 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:53:08.883045 (+ 69us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:53:08.883244 (+ 199us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:53:08.884069 (+ 825us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:53:08.884747 (+ 678us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:53:08.886347 (+ 1600us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:53:08.887101 (+ 754us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:53:08.887264 (+ 163us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:53:08.892073 (+ 4809us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 14:53:08.892080 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 14:53:08.892085 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 14:53:08.892136 (+ 51us) server_negotiation.cc:922] calling sasl_server_start()
0620 14:53:08.895783 (+ 3647us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:53:08.897136 (+ 1353us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:53:08.897140 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:53:08.897143 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:53:08.897310 (+ 167us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:53:08.897976 (+ 666us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:53:08.897981 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:53:08.897984 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:53:08.898319 (+ 335us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 14:53:08.899918 (+ 1599us) server_negotiation.cc:1035] Waiting for connection context
0620 14:53:08.900047 (+ 129us) server_negotiation.cc:299] Negotiation successful
0620 14:53:08.900670 (+ 623us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":27,"server-negotiator.queue_time_us":56}
I20260620 14:53:08.901190 23014 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.97.60" port: 33721 } has no permanent_uuid. Determining permanent_uuid...
I20260620 14:53:08.908205 22865 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 9e796342088f415ba80b5bdb96cd6ff6: Bootstrap starting.
Jun 20 14:53:08 dist-test-slave-q9j6 krb5kdc[22785](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781967188, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.60@KRBTEST.COM for kudu/127.0.97.60@KRBTEST.COM
Jun 20 14:53:08 dist-test-slave-q9j6 krb5kdc[22785](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781967187, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.61@KRBTEST.COM
I20260620 14:53:08.911708 22865 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 9e796342088f415ba80b5bdb96cd6ff6: Neither blocks nor log segments found. Creating new log.
I20260620 14:53:08.913053 22865 log.cc:824] T 00000000000000000000000000000000 P 9e796342088f415ba80b5bdb96cd6ff6: Log is configured to *not* fsync() on all Append() calls
I20260620 14:53:08.917527 22865 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 9e796342088f415ba80b5bdb96cd6ff6: No bootstrap required, opened a new log
I20260620 14:53:08.919314 23019 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:53:08.901976 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.60:33721 (local address 127.0.0.1:37006)
0620 14:53:08.902736 (+ 760us) negotiation.cc:107] Waiting for socket to connect
0620 14:53:08.902784 (+ 48us) client_negotiation.cc:174] Beginning negotiation
0620 14:53:08.902892 (+ 108us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 14:53:08.903215 (+ 323us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 14:53:08.903220 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 14:53:08.903737 (+ 517us) client_negotiation.cc:189] Negotiated authn=SASL
0620 14:53:08.903942 (+ 205us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:53:08.903954 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:53:08.905411 (+ 1457us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:53:08.905417 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:53:08.906550 (+ 1133us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:53:08.906566 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:53:08.907537 (+ 971us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:53:08.907546 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:53:08.907863 (+ 317us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:53:08.908561 (+ 698us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 14:53:08.908600 (+ 39us) client_negotiation.cc:656] calling sasl_client_start()
0620 14:53:08.912237 (+ 3637us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 14:53:08.916219 (+ 3982us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 14:53:08.916225 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 14:53:08.916229 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0620 14:53:08.916630 (+ 401us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 14:53:08.917025 (+ 395us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 14:53:08.917032 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 14:53:08.917035 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0620 14:53:08.917187 (+ 152us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 14:53:08.918179 (+ 992us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 14:53:08.918185 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 14:53:08.918356 (+ 171us) client_negotiation.cc:769] Sending connection context
0620 14:53:08.918560 (+ 204us) client_negotiation.cc:240] Negotiation successful
0620 14:53:08.918906 (+ 346us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":684}
I20260620 14:53:08.919381 23017 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:53:08.902185 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37006 (local address 127.0.97.60:33721)
0620 14:53:08.902360 (+ 175us) server_negotiation.cc:206] Beginning negotiation
0620 14:53:08.902366 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 14:53:08.902996 (+ 630us) server_negotiation.cc:407] Connection header received
0620 14:53:08.903034 (+ 38us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:53:08.903039 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:53:08.903096 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:53:08.904085 (+ 989us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:53:08.904199 (+ 114us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:53:08.905230 (+ 1031us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:53:08.906724 (+ 1494us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:53:08.907332 (+ 608us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:53:08.907467 (+ 135us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:53:08.912531 (+ 5064us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 14:53:08.912539 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 14:53:08.912543 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 14:53:08.912590 (+ 47us) server_negotiation.cc:922] calling sasl_server_start()
0620 14:53:08.916053 (+ 3463us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:53:08.916764 (+ 711us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:53:08.916771 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:53:08.916774 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:53:08.916904 (+ 130us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:53:08.917334 (+ 430us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:53:08.917340 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:53:08.917343 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:53:08.918010 (+ 667us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 14:53:08.918214 (+ 204us) server_negotiation.cc:1035] Waiting for connection context
0620 14:53:08.918653 (+ 439us) server_negotiation.cc:299] Negotiation successful
0620 14:53:08.919031 (+ 378us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":170,"server-negotiator.queue_time_us":53}
I20260620 14:53:08.921962 23021 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:53:08.898519 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41574 (local address 127.0.97.61:45529)
0620 14:53:08.900176 (+ 1657us) server_negotiation.cc:206] Beginning negotiation
0620 14:53:08.900183 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0620 14:53:08.900641 (+ 458us) server_negotiation.cc:407] Connection header received
0620 14:53:08.900819 (+ 178us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:53:08.900827 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:53:08.900906 (+ 79us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:53:08.901024 (+ 118us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:53:08.903010 (+ 1986us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:53:08.904092 (+ 1082us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:53:08.905445 (+ 1353us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:53:08.906115 (+ 670us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:53:08.906248 (+ 133us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:53:08.912878 (+ 6630us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 14:53:08.912886 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 14:53:08.912890 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 14:53:08.912929 (+ 39us) server_negotiation.cc:922] calling sasl_server_start()
0620 14:53:08.916898 (+ 3969us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:53:08.917770 (+ 872us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:53:08.917777 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:53:08.917780 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:53:08.917930 (+ 150us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:53:08.918468 (+ 538us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:53:08.918474 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:53:08.918478 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:53:08.918873 (+ 395us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 14:53:08.921024 (+ 2151us) server_negotiation.cc:1035] Waiting for connection context
0620 14:53:08.921187 (+ 163us) server_negotiation.cc:299] Negotiation successful
0620 14:53:08.921472 (+ 285us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1510,"thread_start_us":1478,"threads_started":1}
I20260620 14:53:08.925038 23014 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c261bc77236444b2a4df214340c6afde: Bootstrap starting.
I20260620 14:53:08.928045 388 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260620 14:53:08.928860 22865 raft_consensus.cc:348] T 00000000000000000000000000000000 P 9e796342088f415ba80b5bdb96cd6ff6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9e796342088f415ba80b5bdb96cd6ff6" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 45139 } } peers { permanent_uuid: "3ef16e7478cd4f99b109748e7c962f1e" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 45529 } } peers { permanent_uuid: "c261bc77236444b2a4df214340c6afde" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 33721 } }
I20260620 14:53:08.929212 22865 raft_consensus.cc:729] T 00000000000000000000000000000000 P 9e796342088f415ba80b5bdb96cd6ff6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9e796342088f415ba80b5bdb96cd6ff6, State: Initialized, Role: FOLLOWER
I20260620 14:53:08.929476 23014 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P c261bc77236444b2a4df214340c6afde: Neither blocks nor log segments found. Creating new log.
I20260620 14:53:08.929667 22865 consensus_queue.cc:260] T 00000000000000000000000000000000 P 9e796342088f415ba80b5bdb96cd6ff6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9e796342088f415ba80b5bdb96cd6ff6" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 45139 } } peers { permanent_uuid: "3ef16e7478cd4f99b109748e7c962f1e" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 45529 } } peers { permanent_uuid: "c261bc77236444b2a4df214340c6afde" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 33721 } }
I20260620 14:53:08.930944 23014 log.cc:824] T 00000000000000000000000000000000 P c261bc77236444b2a4df214340c6afde: Log is configured to *not* fsync() on all Append() calls
I20260620 14:53:08.932067 22840 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "00000000000000000000000000000000"
dest_uuid: "9e796342088f415ba80b5bdb96cd6ff6"
from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:42402
I20260620 14:53:08.932049 23023 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9e796342088f415ba80b5bdb96cd6ff6 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9e796342088f415ba80b5bdb96cd6ff6" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 45139 } } peers { permanent_uuid: "3ef16e7478cd4f99b109748e7c962f1e" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 45529 } } peers { permanent_uuid: "c261bc77236444b2a4df214340c6afde" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 33721 } } }
I20260620 14:53:08.932971 23023 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9e796342088f415ba80b5bdb96cd6ff6 [sys.catalog]: This master's current role is: FOLLOWER
I20260620 14:53:08.933710 22865 sys_catalog.cc:564] T 00000000000000000000000000000000 P 9e796342088f415ba80b5bdb96cd6ff6 [sys.catalog]: configured and running, proceeding with master startup.
I20260620 14:53:08.933794 22840 raft_consensus.cc:482] T 00000000000000000000000000000000 P 9e796342088f415ba80b5bdb96cd6ff6 [term 0 FOLLOWER]: Starting forced leader election (received explicit request)
I20260620 14:53:08.934037 22840 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 9e796342088f415ba80b5bdb96cd6ff6 [term 0 FOLLOWER]: Advancing to term 1
I20260620 14:53:08.934976 23014 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c261bc77236444b2a4df214340c6afde: No bootstrap required, opened a new log
I20260620 14:53:08.939224 22840 raft_consensus.cc:504] T 00000000000000000000000000000000 P 9e796342088f415ba80b5bdb96cd6ff6 [term 1 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9e796342088f415ba80b5bdb96cd6ff6" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 45139 } } peers { permanent_uuid: "3ef16e7478cd4f99b109748e7c962f1e" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 45529 } } peers { permanent_uuid: "c261bc77236444b2a4df214340c6afde" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 33721 } }
I20260620 14:53:08.942629 22840 leader_election.cc:288] T 00000000000000000000000000000000 P 9e796342088f415ba80b5bdb96cd6ff6 [CANDIDATE]: Term 1 election: Requested vote from peers 3ef16e7478cd4f99b109748e7c962f1e (127.0.97.61:45529), c261bc77236444b2a4df214340c6afde (127.0.97.60:33721)
I20260620 14:53:08.945052 23014 raft_consensus.cc:348] T 00000000000000000000000000000000 P c261bc77236444b2a4df214340c6afde [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9e796342088f415ba80b5bdb96cd6ff6" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 45139 } } peers { permanent_uuid: "3ef16e7478cd4f99b109748e7c962f1e" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 45529 } } peers { permanent_uuid: "c261bc77236444b2a4df214340c6afde" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 33721 } }
I20260620 14:53:08.945566 23014 raft_consensus.cc:729] T 00000000000000000000000000000000 P c261bc77236444b2a4df214340c6afde [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c261bc77236444b2a4df214340c6afde, State: Initialized, Role: FOLLOWER
I20260620 14:53:08.946142 23014 consensus_queue.cc:260] T 00000000000000000000000000000000 P c261bc77236444b2a4df214340c6afde [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9e796342088f415ba80b5bdb96cd6ff6" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 45139 } } peers { permanent_uuid: "3ef16e7478cd4f99b109748e7c962f1e" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 45529 } } peers { permanent_uuid: "c261bc77236444b2a4df214340c6afde" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 33721 } }
I20260620 14:53:08.948184 23035 sys_catalog.cc:455] T 00000000000000000000000000000000 P c261bc77236444b2a4df214340c6afde [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9e796342088f415ba80b5bdb96cd6ff6" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 45139 } } peers { permanent_uuid: "3ef16e7478cd4f99b109748e7c962f1e" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 45529 } } peers { permanent_uuid: "c261bc77236444b2a4df214340c6afde" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 33721 } } }
I20260620 14:53:08.949146 23035 sys_catalog.cc:458] T 00000000000000000000000000000000 P c261bc77236444b2a4df214340c6afde [sys.catalog]: This master's current role is: FOLLOWER
I20260620 14:53:08.949348 23014 sys_catalog.cc:564] T 00000000000000000000000000000000 P c261bc77236444b2a4df214340c6afde [sys.catalog]: configured and running, proceeding with master startup.
I20260620 14:53:08.949395 22989 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "9e796342088f415ba80b5bdb96cd6ff6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: true dest_uuid: "c261bc77236444b2a4df214340c6afde"
I20260620 14:53:08.949986 22989 raft_consensus.cc:3037] T 00000000000000000000000000000000 P c261bc77236444b2a4df214340c6afde [term 0 FOLLOWER]: Advancing to term 1
I20260620 14:53:08.953639 22989 raft_consensus.cc:2454] T 00000000000000000000000000000000 P c261bc77236444b2a4df214340c6afde [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9e796342088f415ba80b5bdb96cd6ff6 in term 1.
I20260620 14:53:08.954535 22816 leader_election.cc:302] T 00000000000000000000000000000000 P 9e796342088f415ba80b5bdb96cd6ff6 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 9e796342088f415ba80b5bdb96cd6ff6, c261bc77236444b2a4df214340c6afde; no voters:
I20260620 14:53:08.955176 23023 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 9e796342088f415ba80b5bdb96cd6ff6 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 14:53:08.965451 23023 raft_consensus.cc:686] T 00000000000000000000000000000000 P 9e796342088f415ba80b5bdb96cd6ff6 [term 1 LEADER]: Becoming Leader. State: Replica: 9e796342088f415ba80b5bdb96cd6ff6, State: Running, Role: LEADER
I20260620 14:53:08.966293 23023 consensus_queue.cc:237] T 00000000000000000000000000000000 P 9e796342088f415ba80b5bdb96cd6ff6 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9e796342088f415ba80b5bdb96cd6ff6" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 45139 } } peers { permanent_uuid: "3ef16e7478cd4f99b109748e7c962f1e" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 45529 } } peers { permanent_uuid: "c261bc77236444b2a4df214340c6afde" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 33721 } }
W20260620 14:53:08.973914 22819 tablet.cc:2369] T 00000000000000000000000000000000 P 9e796342088f415ba80b5bdb96cd6ff6: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260620 14:53:08.974186 23042 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9e796342088f415ba80b5bdb96cd6ff6 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 9e796342088f415ba80b5bdb96cd6ff6. Latest consensus state: current_term: 1 leader_uuid: "9e796342088f415ba80b5bdb96cd6ff6" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9e796342088f415ba80b5bdb96cd6ff6" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 45139 } } peers { permanent_uuid: "3ef16e7478cd4f99b109748e7c962f1e" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 45529 } } peers { permanent_uuid: "c261bc77236444b2a4df214340c6afde" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 33721 } } }
I20260620 14:53:08.975271 23042 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9e796342088f415ba80b5bdb96cd6ff6 [sys.catalog]: This master's current role is: LEADER
I20260620 14:53:08.982017 22798 master_runner.cc:186] Error getting master registration for 127.0.97.61:45529: OK, instance_id { permanent_uuid: "3ef16e7478cd4f99b109748e7c962f1e" instance_seqno: 1781967188278135 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260620 14:53:08.997509 22798 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260620 14:53:08.999122 23020 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:53:08.953476 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42410 (local address 127.0.97.62:45139)
0620 14:53:08.953709 (+ 233us) server_negotiation.cc:206] Beginning negotiation
0620 14:53:08.953717 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0620 14:53:08.965112 (+ 11395us) server_negotiation.cc:407] Connection header received
0620 14:53:08.965854 (+ 742us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:53:08.965866 (+ 12us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:53:08.966013 (+ 147us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:53:08.966135 (+ 122us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:53:08.969867 (+ 3732us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:53:08.971561 (+ 1694us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:53:08.974655 (+ 3094us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:53:08.975761 (+ 1106us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:53:08.975934 (+ 173us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:53:08.983418 (+ 7484us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 14:53:08.983429 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 14:53:08.983440 (+ 11us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 14:53:08.983506 (+ 66us) server_negotiation.cc:922] calling sasl_server_start()
0620 14:53:08.987988 (+ 4482us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:53:08.989084 (+ 1096us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:53:08.989091 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:53:08.989095 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:53:08.989298 (+ 203us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:53:08.989829 (+ 531us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:53:08.989834 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:53:08.989837 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:53:08.990264 (+ 427us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 14:53:08.990556 (+ 292us) server_negotiation.cc:1035] Waiting for connection context
0620 14:53:08.998066 (+ 7510us) server_negotiation.cc:299] Negotiation successful
0620 14:53:08.998443 (+ 377us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":55}
W20260620 14:53:09.012261 23046 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 9e796342088f415ba80b5bdb96cd6ff6: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260620 14:53:09.012542 23046 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260620 14:53:09.012657 22939 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.97.60" port: 33721 } attempt: 5
I20260620 14:53:09.014740 23021 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:53:08.953721 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41576 (local address 127.0.97.61:45529)
0620 14:53:08.954005 (+ 284us) server_negotiation.cc:206] Beginning negotiation
0620 14:53:08.954015 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0620 14:53:08.972958 (+ 18943us) server_negotiation.cc:407] Connection header received
0620 14:53:08.973893 (+ 935us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:53:08.973901 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:53:08.974001 (+ 100us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:53:08.974085 (+ 84us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:53:08.984890 (+ 10805us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:53:08.986370 (+ 1480us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:53:08.998210 (+ 11840us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:53:08.999179 (+ 969us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:53:08.999317 (+ 138us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:53:09.005031 (+ 5714us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 14:53:09.005040 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 14:53:09.005045 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 14:53:09.005098 (+ 53us) server_negotiation.cc:922] calling sasl_server_start()
0620 14:53:09.009297 (+ 4199us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:53:09.010403 (+ 1106us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:53:09.010410 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:53:09.010414 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:53:09.010597 (+ 183us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:53:09.011082 (+ 485us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:53:09.011088 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:53:09.011091 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:53:09.011828 (+ 737us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 14:53:09.012259 (+ 431us) server_negotiation.cc:1035] Waiting for connection context
0620 14:53:09.013497 (+ 1238us) server_negotiation.cc:299] Negotiation successful
0620 14:53:09.013964 (+ 467us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":82}
I20260620 14:53:09.018330 23050 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 14:53:09.025395 23050 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 14:53:09.027211 23017 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:53:08.976463 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37022 (local address 127.0.97.60:33721)
0620 14:53:08.976709 (+ 246us) server_negotiation.cc:206] Beginning negotiation
0620 14:53:08.976718 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0620 14:53:08.991968 (+ 15250us) server_negotiation.cc:407] Connection header received
0620 14:53:08.992265 (+ 297us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:53:08.992271 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:53:08.992395 (+ 124us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:53:08.992475 (+ 80us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:53:08.994722 (+ 2247us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:53:08.996288 (+ 1566us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:53:09.002050 (+ 5762us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:53:09.002967 (+ 917us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:53:09.003130 (+ 163us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:53:09.012999 (+ 9869us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 14:53:09.013008 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 14:53:09.013013 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 14:53:09.013070 (+ 57us) server_negotiation.cc:922] calling sasl_server_start()
0620 14:53:09.018162 (+ 5092us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:53:09.019807 (+ 1645us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:53:09.019816 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:53:09.019820 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:53:09.020014 (+ 194us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:53:09.020945 (+ 931us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:53:09.020955 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:53:09.020960 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:53:09.021434 (+ 474us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 14:53:09.021729 (+ 295us) server_negotiation.cc:1035] Waiting for connection context
0620 14:53:09.026409 (+ 4680us) server_negotiation.cc:299] Negotiation successful
0620 14:53:09.026746 (+ 337us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":83}
Jun 20 14:53:09 dist-test-slave-q9j6 krb5kdc[22785](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781967188, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.61@KRBTEST.COM for kudu/127.0.97.60@KRBTEST.COM
I20260620 14:53:09.044150 22989 raft_consensus.cc:1261] T 00000000000000000000000000000000 P c261bc77236444b2a4df214340c6afde [term 1 FOLLOWER]: Refusing update from remote peer 9e796342088f415ba80b5bdb96cd6ff6: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 14:53:09.045512 23023 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 9e796342088f415ba80b5bdb96cd6ff6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c261bc77236444b2a4df214340c6afde" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 33721 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 14:53:09.050866 22943 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:53:09.013425 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.60:33721 (local address 127.0.0.1:37036)
0620 14:53:09.013913 (+ 488us) negotiation.cc:107] Waiting for socket to connect
0620 14:53:09.013935 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0620 14:53:09.014070 (+ 135us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 14:53:09.028196 (+ 14126us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 14:53:09.028204 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 14:53:09.028801 (+ 597us) client_negotiation.cc:189] Negotiated authn=SASL
0620 14:53:09.029054 (+ 253us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:53:09.029073 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:53:09.030982 (+ 1909us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:53:09.030991 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:53:09.032662 (+ 1671us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:53:09.032687 (+ 25us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:53:09.034229 (+ 1542us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:53:09.034236 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:53:09.034492 (+ 256us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:53:09.035419 (+ 927us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 14:53:09.035448 (+ 29us) client_negotiation.cc:656] calling sasl_client_start()
0620 14:53:09.039296 (+ 3848us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 14:53:09.046672 (+ 7376us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 14:53:09.046681 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 14:53:09.046685 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0620 14:53:09.047492 (+ 807us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 14:53:09.048516 (+ 1024us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 14:53:09.048522 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 14:53:09.048527 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0620 14:53:09.048771 (+ 244us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 14:53:09.049670 (+ 899us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 14:53:09.049675 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 14:53:09.049863 (+ 188us) client_negotiation.cc:769] Sending connection context
0620 14:53:09.050048 (+ 185us) client_negotiation.cc:240] Negotiation successful
0620 14:53:09.050426 (+ 378us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":318}
I20260620 14:53:09.052207 23017 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:53:09.014337 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37036 (local address 127.0.97.60:33721)
0620 14:53:09.027799 (+ 13462us) server_negotiation.cc:206] Beginning negotiation
0620 14:53:09.027809 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0620 14:53:09.027830 (+ 21us) server_negotiation.cc:407] Connection header received
0620 14:53:09.027907 (+ 77us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:53:09.027913 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:53:09.027998 (+ 85us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:53:09.028127 (+ 129us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:53:09.029569 (+ 1442us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:53:09.030817 (+ 1248us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:53:09.032946 (+ 2129us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:53:09.033748 (+ 802us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:53:09.033902 (+ 154us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:53:09.039697 (+ 5795us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 14:53:09.039706 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 14:53:09.039711 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 14:53:09.039760 (+ 49us) server_negotiation.cc:922] calling sasl_server_start()
0620 14:53:09.046478 (+ 6718us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:53:09.047689 (+ 1211us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:53:09.047696 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:53:09.047700 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:53:09.047881 (+ 181us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:53:09.048942 (+ 1061us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:53:09.048948 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:53:09.048952 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:53:09.049389 (+ 437us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 14:53:09.049577 (+ 188us) server_negotiation.cc:1035] Waiting for connection context
0620 14:53:09.051365 (+ 1788us) server_negotiation.cc:299] Negotiation successful
0620 14:53:09.051717 (+ 352us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":13330,"thread_start_us":19199,"threads_started":1}
I20260620 14:53:09.060062 22939 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 3ef16e7478cd4f99b109748e7c962f1e: Bootstrap starting.
W20260620 14:53:09.071547 22968 tablet.cc:2369] T 00000000000000000000000000000000 P c261bc77236444b2a4df214340c6afde: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260620 14:53:09.075343 22939 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 3ef16e7478cd4f99b109748e7c962f1e: Neither blocks nor log segments found. Creating new log.
I20260620 14:53:09.084364 22939 log.cc:824] T 00000000000000000000000000000000 P 3ef16e7478cd4f99b109748e7c962f1e: Log is configured to *not* fsync() on all Append() calls
I20260620 14:53:09.087569 23035 sys_catalog.cc:455] T 00000000000000000000000000000000 P c261bc77236444b2a4df214340c6afde [sys.catalog]: SysCatalogTable state changed. Reason: New leader 9e796342088f415ba80b5bdb96cd6ff6. Latest consensus state: current_term: 1 leader_uuid: "9e796342088f415ba80b5bdb96cd6ff6" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9e796342088f415ba80b5bdb96cd6ff6" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 45139 } } peers { permanent_uuid: "3ef16e7478cd4f99b109748e7c962f1e" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 45529 } } peers { permanent_uuid: "c261bc77236444b2a4df214340c6afde" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 33721 } } }
I20260620 14:53:09.088037 23035 sys_catalog.cc:458] T 00000000000000000000000000000000 P c261bc77236444b2a4df214340c6afde [sys.catalog]: This master's current role is: FOLLOWER
I20260620 14:53:09.091501 23053 mvcc.cc:205] Tried to move back new op lower bound from 7298937606307004416 to 7298937606012432384. Current Snapshot: MvccSnapshot[applied={T|T < 7298937606307004416}]
I20260620 14:53:09.112670 23023 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9e796342088f415ba80b5bdb96cd6ff6 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "9e796342088f415ba80b5bdb96cd6ff6" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9e796342088f415ba80b5bdb96cd6ff6" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 45139 } } peers { permanent_uuid: "3ef16e7478cd4f99b109748e7c962f1e" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 45529 } } peers { permanent_uuid: "c261bc77236444b2a4df214340c6afde" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 33721 } } }
I20260620 14:53:09.113061 23023 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9e796342088f415ba80b5bdb96cd6ff6 [sys.catalog]: This master's current role is: LEADER
I20260620 14:53:09.114841 23050 catalog_manager.cc:1347] Generated new cluster ID: ce7c821f923e46cc93a341a4b43e096e
I20260620 14:53:09.114996 23050 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 14:53:09.118450 22939 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 3ef16e7478cd4f99b109748e7c962f1e: No bootstrap required, opened a new log
I20260620 14:53:09.116660 22947 master_runner.cc:186] Error getting master registration for 127.0.97.61:45529: OK, instance_id { permanent_uuid: "3ef16e7478cd4f99b109748e7c962f1e" instance_seqno: 1781967188278135 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260620 14:53:09.129431 22947 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260620 14:53:09.135155 23035 sys_catalog.cc:455] T 00000000000000000000000000000000 P c261bc77236444b2a4df214340c6afde [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "9e796342088f415ba80b5bdb96cd6ff6" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9e796342088f415ba80b5bdb96cd6ff6" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 45139 } } peers { permanent_uuid: "3ef16e7478cd4f99b109748e7c962f1e" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 45529 } } peers { permanent_uuid: "c261bc77236444b2a4df214340c6afde" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 33721 } } }
I20260620 14:53:09.135495 23035 sys_catalog.cc:458] T 00000000000000000000000000000000 P c261bc77236444b2a4df214340c6afde [sys.catalog]: This master's current role is: FOLLOWER
I20260620 14:53:09.137599 22939 raft_consensus.cc:348] T 00000000000000000000000000000000 P 3ef16e7478cd4f99b109748e7c962f1e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9e796342088f415ba80b5bdb96cd6ff6" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 45139 } } peers { permanent_uuid: "3ef16e7478cd4f99b109748e7c962f1e" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 45529 } } peers { permanent_uuid: "c261bc77236444b2a4df214340c6afde" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 33721 } }
I20260620 14:53:09.138113 22939 raft_consensus.cc:729] T 00000000000000000000000000000000 P 3ef16e7478cd4f99b109748e7c962f1e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3ef16e7478cd4f99b109748e7c962f1e, State: Initialized, Role: FOLLOWER
I20260620 14:53:09.138723 22939 consensus_queue.cc:260] T 00000000000000000000000000000000 P 3ef16e7478cd4f99b109748e7c962f1e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9e796342088f415ba80b5bdb96cd6ff6" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 45139 } } peers { permanent_uuid: "3ef16e7478cd4f99b109748e7c962f1e" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 45529 } } peers { permanent_uuid: "c261bc77236444b2a4df214340c6afde" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 33721 } }
I20260620 14:53:09.149140 23050 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 14:53:09.151237 23050 catalog_manager.cc:1504] Loading token signing keys...
I20260620 14:53:09.154124 23063 catalog_manager.cc:1259] Loaded cluster ID: ce7c821f923e46cc93a341a4b43e096e
I20260620 14:53:09.154664 23063 catalog_manager.cc:1552] T 00000000000000000000000000000000 P c261bc77236444b2a4df214340c6afde: loading cluster ID for follower catalog manager: success
I20260620 14:53:09.156960 22939 sys_catalog.cc:564] T 00000000000000000000000000000000 P 3ef16e7478cd4f99b109748e7c962f1e [sys.catalog]: configured and running, proceeding with master startup.
I20260620 14:53:09.157285 22914 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "9e796342088f415ba80b5bdb96cd6ff6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: true dest_uuid: "3ef16e7478cd4f99b109748e7c962f1e"
I20260620 14:53:09.157559 22913 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 3ef16e7478cd4f99b109748e7c962f1e [term 0 FOLLOWER]: Advancing to term 1
I20260620 14:53:09.164296 22913 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 3ef16e7478cd4f99b109748e7c962f1e [term 1 FOLLOWER]: Refusing update from remote peer 9e796342088f415ba80b5bdb96cd6ff6: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 14:53:09.165208 23067 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3ef16e7478cd4f99b109748e7c962f1e [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9e796342088f415ba80b5bdb96cd6ff6" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 45139 } } peers { permanent_uuid: "3ef16e7478cd4f99b109748e7c962f1e" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 45529 } } peers { permanent_uuid: "c261bc77236444b2a4df214340c6afde" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 33721 } } }
I20260620 14:53:09.165747 23067 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3ef16e7478cd4f99b109748e7c962f1e [sys.catalog]: This master's current role is: FOLLOWER
I20260620 14:53:09.166774 23063 catalog_manager.cc:1574] T 00000000000000000000000000000000 P c261bc77236444b2a4df214340c6afde: acquiring CA information for follower catalog manager: success
W20260620 14:53:09.168442 23063 catalog_manager.cc:873] Not found: no TSK found in the system table: failed to prepare follower catalog manager, will retry
I20260620 14:53:09.169305 23042 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 9e796342088f415ba80b5bdb96cd6ff6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3ef16e7478cd4f99b109748e7c962f1e" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 45529 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 14:53:09.179382 23050 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 9e796342088f415ba80b5bdb96cd6ff6: Generated new TSK 0
I20260620 14:53:09.180646 23050 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260620 14:53:09.231163 22914 raft_consensus.cc:2396] T 00000000000000000000000000000000 P 3ef16e7478cd4f99b109748e7c962f1e [term 1 FOLLOWER]: Leader election vote request: Denying vote to candidate 9e796342088f415ba80b5bdb96cd6ff6 for term 1 because replica has last-logged OpId of term: 1 index: 4, which is greater than that of the candidate, which has last-logged OpId of term: 0 index: 0.
I20260620 14:53:09.231906 23067 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3ef16e7478cd4f99b109748e7c962f1e [sys.catalog]: SysCatalogTable state changed. Reason: New leader 9e796342088f415ba80b5bdb96cd6ff6. Latest consensus state: current_term: 1 leader_uuid: "9e796342088f415ba80b5bdb96cd6ff6" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9e796342088f415ba80b5bdb96cd6ff6" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 45139 } } peers { permanent_uuid: "3ef16e7478cd4f99b109748e7c962f1e" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 45529 } } peers { permanent_uuid: "c261bc77236444b2a4df214340c6afde" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 33721 } } }
I20260620 14:53:09.232213 23067 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3ef16e7478cd4f99b109748e7c962f1e [sys.catalog]: This master's current role is: FOLLOWER
I20260620 14:53:09.232591 23076 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3ef16e7478cd4f99b109748e7c962f1e [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "9e796342088f415ba80b5bdb96cd6ff6" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9e796342088f415ba80b5bdb96cd6ff6" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 45139 } } peers { permanent_uuid: "3ef16e7478cd4f99b109748e7c962f1e" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 45529 } } peers { permanent_uuid: "c261bc77236444b2a4df214340c6afde" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 33721 } } }
I20260620 14:53:09.232934 23076 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3ef16e7478cd4f99b109748e7c962f1e [sys.catalog]: This master's current role is: FOLLOWER
I20260620 14:53:09.232806 23042 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9e796342088f415ba80b5bdb96cd6ff6 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "9e796342088f415ba80b5bdb96cd6ff6" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9e796342088f415ba80b5bdb96cd6ff6" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 45139 } } peers { permanent_uuid: "3ef16e7478cd4f99b109748e7c962f1e" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 45529 } } peers { permanent_uuid: "c261bc77236444b2a4df214340c6afde" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 33721 } } }
I20260620 14:53:09.233127 23042 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9e796342088f415ba80b5bdb96cd6ff6 [sys.catalog]: This master's current role is: LEADER
I20260620 14:53:09.256162 23084 catalog_manager.cc:1259] Loaded cluster ID: ce7c821f923e46cc93a341a4b43e096e
I20260620 14:53:09.256397 23084 catalog_manager.cc:1552] T 00000000000000000000000000000000 P 3ef16e7478cd4f99b109748e7c962f1e: loading cluster ID for follower catalog manager: success
I20260620 14:53:09.260751 23084 catalog_manager.cc:1574] T 00000000000000000000000000000000 P 3ef16e7478cd4f99b109748e7c962f1e: acquiring CA information for follower catalog manager: success
I20260620 14:53:09.266656 23084 catalog_manager.cc:1602] T 00000000000000000000000000000000 P 3ef16e7478cd4f99b109748e7c962f1e: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
W20260620 14:53:09.344794 23020 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 14:53:09.333017 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42424 (local address 127.0.97.62:45139)
0620 14:53:09.333266 (+ 249us) server_negotiation.cc:206] Beginning negotiation
0620 14:53:09.333274 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0620 14:53:09.342482 (+ 9208us) server_negotiation.cc:407] Connection header received
0620 14:53:09.342836 (+ 354us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:53:09.342843 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:53:09.342945 (+ 102us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:53:09.343025 (+ 80us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:53:09.344483 (+ 1458us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:42424: BlockingRecv error: recv got EOF from 127.0.0.1:42424 (error 108)
Metrics: {"server-negotiator.queue_time_us":73}
W20260620 14:53:09.344938 23021 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 14:53:09.335292 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41584 (local address 127.0.97.61:45529)
0620 14:53:09.335557 (+ 265us) server_negotiation.cc:206] Beginning negotiation
0620 14:53:09.335566 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0620 14:53:09.342685 (+ 7119us) server_negotiation.cc:407] Connection header received
0620 14:53:09.342968 (+ 283us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:53:09.342975 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:53:09.343088 (+ 113us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:53:09.343174 (+ 86us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:53:09.344637 (+ 1463us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:41584: BlockingRecv error: recv got EOF from 127.0.0.1:41584 (error 108)
Metrics: {"server-negotiator.queue_time_us":93}
W20260620 14:53:09.357646 23017 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 14:53:09.341817 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37048 (local address 127.0.97.60:33721)
0620 14:53:09.342054 (+ 237us) server_negotiation.cc:206] Beginning negotiation
0620 14:53:09.342061 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0620 14:53:09.345765 (+ 3704us) server_negotiation.cc:407] Connection header received
0620 14:53:09.355857 (+ 10092us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:53:09.355867 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:53:09.355997 (+ 130us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:53:09.356100 (+ 103us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:53:09.357332 (+ 1232us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:37048: BlockingRecv error: recv got EOF from 127.0.0.1:37048 (error 108)
Metrics: {"server-negotiator.queue_time_us":58}
I20260620 14:53:09.417732 23021 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:53:09.393061 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41586 (local address 127.0.97.61:45529)
0620 14:53:09.393258 (+ 197us) server_negotiation.cc:206] Beginning negotiation
0620 14:53:09.393265 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0620 14:53:09.408360 (+ 15095us) server_negotiation.cc:407] Connection header received
0620 14:53:09.408608 (+ 248us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:53:09.408616 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:53:09.408896 (+ 280us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:53:09.409083 (+ 187us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 14:53:09.409864 (+ 781us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:53:09.411152 (+ 1288us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:53:09.413482 (+ 2330us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:53:09.414298 (+ 816us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:53:09.414448 (+ 150us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:53:09.415616 (+ 1168us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 14:53:09.416427 (+ 811us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0620 14:53:09.416593 (+ 166us) server_negotiation.cc:1035] Waiting for connection context
0620 14:53:09.417202 (+ 609us) server_negotiation.cc:299] Negotiation successful
0620 14:53:09.417310 (+ 108us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":71}
W20260620 14:53:09.467101 23017 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 14:53:09.456641 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55044 (local address 127.0.97.60:33721)
0620 14:53:09.456890 (+ 249us) server_negotiation.cc:206] Beginning negotiation
0620 14:53:09.456897 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0620 14:53:09.464565 (+ 7668us) server_negotiation.cc:407] Connection header received
0620 14:53:09.465221 (+ 656us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:53:09.465229 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:53:09.465392 (+ 163us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:53:09.465494 (+ 102us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:53:09.466787 (+ 1293us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:55044: BlockingRecv error: recv got EOF from 127.0.0.1:55044 (error 108)
Metrics: {"server-negotiator.queue_time_us":81}
I20260620 14:53:09.528395 23021 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:53:09.498661 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41902 (local address 127.0.97.61:45529)
0620 14:53:09.498967 (+ 306us) server_negotiation.cc:206] Beginning negotiation
0620 14:53:09.498978 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0620 14:53:09.520393 (+ 21415us) server_negotiation.cc:407] Connection header received
0620 14:53:09.520686 (+ 293us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:53:09.520693 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:53:09.520793 (+ 100us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:53:09.520899 (+ 106us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 14:53:09.521626 (+ 727us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:53:09.522914 (+ 1288us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:53:09.525328 (+ 2414us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:53:09.526157 (+ 829us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:53:09.526353 (+ 196us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:53:09.527060 (+ 707us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 14:53:09.527237 (+ 177us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0620 14:53:09.527330 (+ 93us) server_negotiation.cc:1035] Waiting for connection context
0620 14:53:09.527920 (+ 590us) server_negotiation.cc:299] Negotiation successful
0620 14:53:09.528018 (+ 98us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":88}
W20260620 14:53:09.574682 23017 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 14:53:09.566017 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55060 (local address 127.0.97.60:33721)
0620 14:53:09.566243 (+ 226us) server_negotiation.cc:206] Beginning negotiation
0620 14:53:09.566251 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0620 14:53:09.572391 (+ 6140us) server_negotiation.cc:407] Connection header received
0620 14:53:09.572672 (+ 281us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:53:09.572679 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:53:09.572785 (+ 106us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:53:09.572869 (+ 84us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:53:09.574436 (+ 1567us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:55060: BlockingRecv error: recv got EOF from 127.0.0.1:55060 (error 108)
Metrics: {"server-negotiator.queue_time_us":87}
I20260620 14:53:09.630182 23021 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:53:09.614196 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41914 (local address 127.0.97.61:45529)
0620 14:53:09.614470 (+ 274us) server_negotiation.cc:206] Beginning negotiation
0620 14:53:09.614477 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0620 14:53:09.622902 (+ 8425us) server_negotiation.cc:407] Connection header received
0620 14:53:09.623234 (+ 332us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:53:09.623240 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:53:09.623337 (+ 97us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:53:09.623404 (+ 67us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 14:53:09.624157 (+ 753us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:53:09.625338 (+ 1181us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:53:09.627577 (+ 2239us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:53:09.628369 (+ 792us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:53:09.628490 (+ 121us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:53:09.629251 (+ 761us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 14:53:09.629431 (+ 180us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0620 14:53:09.629498 (+ 67us) server_negotiation.cc:1035] Waiting for connection context
0620 14:53:09.629740 (+ 242us) server_negotiation.cc:299] Negotiation successful
0620 14:53:09.629842 (+ 102us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":85}
W20260620 14:53:09.682753 23017 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 14:53:09.672735 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55074 (local address 127.0.97.60:33721)
0620 14:53:09.672950 (+ 215us) server_negotiation.cc:206] Beginning negotiation
0620 14:53:09.672958 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0620 14:53:09.680538 (+ 7580us) server_negotiation.cc:407] Connection header received
0620 14:53:09.681101 (+ 563us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:53:09.681111 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:53:09.681212 (+ 101us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:53:09.681287 (+ 75us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:53:09.682462 (+ 1175us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:55074: BlockingRecv error: recv got EOF from 127.0.0.1:55074 (error 108)
Metrics: {"server-negotiator.queue_time_us":71}
I20260620 14:53:09.725970 23021 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:53:09.718140 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41928 (local address 127.0.97.61:45529)
0620 14:53:09.718358 (+ 218us) server_negotiation.cc:206] Beginning negotiation
0620 14:53:09.718369 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0620 14:53:09.718984 (+ 615us) server_negotiation.cc:407] Connection header received
0620 14:53:09.719206 (+ 222us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:53:09.719215 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:53:09.719322 (+ 107us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:53:09.719445 (+ 123us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 14:53:09.720024 (+ 579us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:53:09.721314 (+ 1290us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:53:09.723471 (+ 2157us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:53:09.724279 (+ 808us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:53:09.724463 (+ 184us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:53:09.725000 (+ 537us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 14:53:09.725216 (+ 216us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0620 14:53:09.725328 (+ 112us) server_negotiation.cc:1035] Waiting for connection context
0620 14:53:09.725546 (+ 218us) server_negotiation.cc:299] Negotiation successful
0620 14:53:09.725645 (+ 99us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":80}
W20260620 14:53:09.737478 23017 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 14:53:09.734624 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55080 (local address 127.0.97.60:33721)
0620 14:53:09.734948 (+ 324us) server_negotiation.cc:206] Beginning negotiation
0620 14:53:09.734955 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0620 14:53:09.735484 (+ 529us) server_negotiation.cc:407] Connection header received
0620 14:53:09.735785 (+ 301us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:53:09.735791 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:53:09.735892 (+ 101us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:53:09.736021 (+ 129us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:53:09.737206 (+ 1185us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:55080: BlockingRecv error: recv got EOF from 127.0.0.1:55080 (error 108)
Metrics: {"server-negotiator.queue_time_us":143}
I20260620 14:53:09.777787 23021 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:53:09.770010 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41936 (local address 127.0.97.61:45529)
0620 14:53:09.770210 (+ 200us) server_negotiation.cc:206] Beginning negotiation
0620 14:53:09.770218 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0620 14:53:09.770856 (+ 638us) server_negotiation.cc:407] Connection header received
0620 14:53:09.771115 (+ 259us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:53:09.771120 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:53:09.771249 (+ 129us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:53:09.771364 (+ 115us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 14:53:09.771906 (+ 542us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:53:09.773111 (+ 1205us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:53:09.775213 (+ 2102us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:53:09.775948 (+ 735us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:53:09.776103 (+ 155us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:53:09.776762 (+ 659us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 14:53:09.776933 (+ 171us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0620 14:53:09.777013 (+ 80us) server_negotiation.cc:1035] Waiting for connection context
0620 14:53:09.777302 (+ 289us) server_negotiation.cc:299] Negotiation successful
0620 14:53:09.777404 (+ 102us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":77}
W20260620 14:53:09.790210 23017 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 14:53:09.785176 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55092 (local address 127.0.97.60:33721)
0620 14:53:09.785371 (+ 195us) server_negotiation.cc:206] Beginning negotiation
0620 14:53:09.785378 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0620 14:53:09.788379 (+ 3001us) server_negotiation.cc:407] Connection header received
0620 14:53:09.788611 (+ 232us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:53:09.788617 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:53:09.788728 (+ 111us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:53:09.788857 (+ 129us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:53:09.789887 (+ 1030us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:55092: BlockingRecv error: recv got EOF from 127.0.0.1:55092 (error 108)
Metrics: {"server-negotiator.queue_time_us":66}
I20260620 14:53:09.817442 23021 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:53:09.809925 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41952 (local address 127.0.97.61:45529)
0620 14:53:09.810127 (+ 202us) server_negotiation.cc:206] Beginning negotiation
0620 14:53:09.810135 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0620 14:53:09.810749 (+ 614us) server_negotiation.cc:407] Connection header received
0620 14:53:09.810965 (+ 216us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:53:09.810973 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:53:09.811083 (+ 110us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:53:09.811197 (+ 114us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 14:53:09.811773 (+ 576us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:53:09.812959 (+ 1186us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:53:09.815064 (+ 2105us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:53:09.815778 (+ 714us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:53:09.815942 (+ 164us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:53:09.816538 (+ 596us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 14:53:09.816698 (+ 160us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0620 14:53:09.816760 (+ 62us) server_negotiation.cc:1035] Waiting for connection context
0620 14:53:09.817021 (+ 261us) server_negotiation.cc:299] Negotiation successful
0620 14:53:09.817118 (+ 97us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":73}
W20260620 14:53:09.828796 23017 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 14:53:09.826073 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55102 (local address 127.0.97.60:33721)
0620 14:53:09.826255 (+ 182us) server_negotiation.cc:206] Beginning negotiation
0620 14:53:09.826263 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0620 14:53:09.826917 (+ 654us) server_negotiation.cc:407] Connection header received
0620 14:53:09.827150 (+ 233us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:53:09.827158 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:53:09.827291 (+ 133us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:53:09.827440 (+ 149us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:53:09.828499 (+ 1059us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:55102: BlockingRecv error: recv got EOF from 127.0.0.1:55102 (error 108)
Metrics: {"server-negotiator.queue_time_us":66}
I20260620 14:53:09.879299 23021 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:53:09.873612 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41962 (local address 127.0.97.61:45529)
0620 14:53:09.873827 (+ 215us) server_negotiation.cc:206] Beginning negotiation
0620 14:53:09.873835 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0620 14:53:09.874222 (+ 387us) server_negotiation.cc:407] Connection header received
0620 14:53:09.874469 (+ 247us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:53:09.874474 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:53:09.874547 (+ 73us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:53:09.874656 (+ 109us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 14:53:09.875077 (+ 421us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:53:09.875984 (+ 907us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:53:09.877380 (+ 1396us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:53:09.877807 (+ 427us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:53:09.878080 (+ 273us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:53:09.878501 (+ 421us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 14:53:09.878634 (+ 133us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0620 14:53:09.878740 (+ 106us) server_negotiation.cc:1035] Waiting for connection context
0620 14:53:09.878897 (+ 157us) server_negotiation.cc:299] Negotiation successful
0620 14:53:09.878985 (+ 88us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":65}
W20260620 14:53:09.892587 23017 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 14:53:09.890506 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55106 (local address 127.0.97.60:33721)
0620 14:53:09.890765 (+ 259us) server_negotiation.cc:206] Beginning negotiation
0620 14:53:09.890774 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0620 14:53:09.891062 (+ 288us) server_negotiation.cc:407] Connection header received
0620 14:53:09.891229 (+ 167us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:53:09.891235 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:53:09.891352 (+ 117us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:53:09.891496 (+ 144us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:53:09.892314 (+ 818us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:55106: BlockingRecv error: recv got EOF from 127.0.0.1:55106 (error 108)
Metrics: {"server-negotiator.queue_time_us":98}
I20260620 14:53:09.965855 23021 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:53:09.960338 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41976 (local address 127.0.97.61:45529)
0620 14:53:09.960597 (+ 259us) server_negotiation.cc:206] Beginning negotiation
0620 14:53:09.960603 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 14:53:09.960911 (+ 308us) server_negotiation.cc:407] Connection header received
0620 14:53:09.961143 (+ 232us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:53:09.961148 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:53:09.961229 (+ 81us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:53:09.961331 (+ 102us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 14:53:09.961780 (+ 449us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:53:09.962580 (+ 800us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:53:09.964059 (+ 1479us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:53:09.964557 (+ 498us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:53:09.964669 (+ 112us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:53:09.965041 (+ 372us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 14:53:09.965167 (+ 126us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0620 14:53:09.965241 (+ 74us) server_negotiation.cc:1035] Waiting for connection context
0620 14:53:09.965383 (+ 142us) server_negotiation.cc:299] Negotiation successful
0620 14:53:09.965460 (+ 77us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":110}
W20260620 14:53:09.974083 23017 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 14:53:09.972211 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55112 (local address 127.0.97.60:33721)
0620 14:53:09.972407 (+ 196us) server_negotiation.cc:206] Beginning negotiation
0620 14:53:09.972412 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 14:53:09.972843 (+ 431us) server_negotiation.cc:407] Connection header received
0620 14:53:09.972985 (+ 142us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:53:09.972990 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:53:09.973069 (+ 79us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:53:09.973165 (+ 96us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:53:09.973888 (+ 723us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:55112: BlockingRecv error: recv got EOF from 127.0.0.1:55112 (error 108)
Metrics: {"server-negotiator.queue_time_us":64}
I20260620 14:53:10.111389 23021 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:53:10.105879 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41990 (local address 127.0.97.61:45529)
0620 14:53:10.106136 (+ 257us) server_negotiation.cc:206] Beginning negotiation
0620 14:53:10.106146 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0620 14:53:10.106523 (+ 377us) server_negotiation.cc:407] Connection header received
0620 14:53:10.106760 (+ 237us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:53:10.106764 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:53:10.106836 (+ 72us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:53:10.106945 (+ 109us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 14:53:10.107413 (+ 468us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:53:10.108187 (+ 774us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:53:10.109589 (+ 1402us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:53:10.110021 (+ 432us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:53:10.110165 (+ 144us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:53:10.110630 (+ 465us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 14:53:10.110745 (+ 115us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0620 14:53:10.110849 (+ 104us) server_negotiation.cc:1035] Waiting for connection context
0620 14:53:10.111013 (+ 164us) server_negotiation.cc:299] Negotiation successful
0620 14:53:10.111102 (+ 89us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":80}
W20260620 14:53:10.119516 23017 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 14:53:10.117562 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55122 (local address 127.0.97.60:33721)
0620 14:53:10.117753 (+ 191us) server_negotiation.cc:206] Beginning negotiation
0620 14:53:10.117758 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 14:53:10.118201 (+ 443us) server_negotiation.cc:407] Connection header received
0620 14:53:10.118345 (+ 144us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:53:10.118350 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:53:10.118462 (+ 112us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:53:10.118576 (+ 114us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:53:10.119308 (+ 732us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:55122: BlockingRecv error: recv got EOF from 127.0.0.1:55122 (error 108)
Metrics: {"server-negotiator.queue_time_us":67}
I20260620 14:53:10.171217 23063 catalog_manager.cc:1602] T 00000000000000000000000000000000 P c261bc77236444b2a4df214340c6afde: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260620 14:53:10.391206 23021 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:53:10.385672 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41996 (local address 127.0.97.61:45529)
0620 14:53:10.385946 (+ 274us) server_negotiation.cc:206] Beginning negotiation
0620 14:53:10.385955 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0620 14:53:10.386185 (+ 230us) server_negotiation.cc:407] Connection header received
0620 14:53:10.386450 (+ 265us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:53:10.386455 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:53:10.386543 (+ 88us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:53:10.386659 (+ 116us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 14:53:10.387126 (+ 467us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:53:10.387885 (+ 759us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:53:10.389365 (+ 1480us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:53:10.389782 (+ 417us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:53:10.389940 (+ 158us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:53:10.390391 (+ 451us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 14:53:10.390513 (+ 122us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0620 14:53:10.390621 (+ 108us) server_negotiation.cc:1035] Waiting for connection context
0620 14:53:10.390782 (+ 161us) server_negotiation.cc:299] Negotiation successful
0620 14:53:10.390872 (+ 90us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":106}
I20260620 14:53:10.404174 23017 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:53:10.397784 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55136 (local address 127.0.97.60:33721)
0620 14:53:10.398022 (+ 238us) server_negotiation.cc:206] Beginning negotiation
0620 14:53:10.398028 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 14:53:10.398353 (+ 325us) server_negotiation.cc:407] Connection header received
0620 14:53:10.398596 (+ 243us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:53:10.398602 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:53:10.398839 (+ 237us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:53:10.398969 (+ 130us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 14:53:10.399470 (+ 501us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:53:10.400252 (+ 782us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:53:10.401706 (+ 1454us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:53:10.402117 (+ 411us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:53:10.402267 (+ 150us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:53:10.402864 (+ 597us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 14:53:10.403424 (+ 560us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0620 14:53:10.403667 (+ 243us) server_negotiation.cc:1035] Waiting for connection context
0620 14:53:10.403724 (+ 57us) server_negotiation.cc:299] Negotiation successful
0620 14:53:10.403816 (+ 92us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":70}
I20260620 14:53:10.419457 23021 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:53:10.411016 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42002 (local address 127.0.97.61:45529)
0620 14:53:10.411218 (+ 202us) server_negotiation.cc:206] Beginning negotiation
0620 14:53:10.411225 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0620 14:53:10.412055 (+ 830us) server_negotiation.cc:407] Connection header received
0620 14:53:10.412399 (+ 344us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:53:10.412417 (+ 18us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:53:10.412540 (+ 123us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:53:10.412630 (+ 90us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 14:53:10.413692 (+ 1062us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:53:10.415106 (+ 1414us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:53:10.417378 (+ 2272us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:53:10.418073 (+ 695us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:53:10.418235 (+ 162us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:53:10.418673 (+ 438us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 14:53:10.418784 (+ 111us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0620 14:53:10.418887 (+ 103us) server_negotiation.cc:1035] Waiting for connection context
0620 14:53:10.419055 (+ 168us) server_negotiation.cc:299] Negotiation successful
0620 14:53:10.419142 (+ 87us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":71}
I20260620 14:53:10.420991 23200 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:53:10.411157 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45780 (local address 127.0.97.62:45139)
0620 14:53:10.413997 (+ 2840us) server_negotiation.cc:206] Beginning negotiation
0620 14:53:10.414001 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 14:53:10.414020 (+ 19us) server_negotiation.cc:407] Connection header received
0620 14:53:10.414118 (+ 98us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:53:10.414121 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:53:10.414291 (+ 170us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:53:10.414500 (+ 209us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 14:53:10.415239 (+ 739us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:53:10.416255 (+ 1016us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:53:10.418118 (+ 1863us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:53:10.418766 (+ 648us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:53:10.418928 (+ 162us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:53:10.419969 (+ 1041us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 14:53:10.420346 (+ 377us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0620 14:53:10.420425 (+ 79us) server_negotiation.cc:1035] Waiting for connection context
0620 14:53:10.420629 (+ 204us) server_negotiation.cc:299] Negotiation successful
0620 14:53:10.420717 (+ 88us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":2729,"thread_start_us":2623,"threads_started":1}
I20260620 14:53:10.421521 23017 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:53:10.412364 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55140 (local address 127.0.97.60:33721)
0620 14:53:10.412565 (+ 201us) server_negotiation.cc:206] Beginning negotiation
0620 14:53:10.412569 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 14:53:10.415346 (+ 2777us) server_negotiation.cc:407] Connection header received
0620 14:53:10.415533 (+ 187us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:53:10.415538 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:53:10.415655 (+ 117us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:53:10.415752 (+ 97us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 14:53:10.416471 (+ 719us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:53:10.417489 (+ 1018us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:53:10.419397 (+ 1908us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:53:10.420021 (+ 624us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:53:10.420123 (+ 102us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:53:10.420767 (+ 644us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 14:53:10.420893 (+ 126us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0620 14:53:10.420958 (+ 65us) server_negotiation.cc:1035] Waiting for connection context
0620 14:53:10.421213 (+ 255us) server_negotiation.cc:299] Negotiation successful
0620 14:53:10.421297 (+ 84us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":52}
I20260620 14:53:10.426474 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskUci3DU/build/asan/bin/kudu pid 22798
I20260620 14:53:10.427589 22860 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 14:53:10.540153 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskUci3DU/build/asan/bin/kudu with pid 22798
I20260620 14:53:10.562266 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskUci3DU/build/asan/bin/kudu pid 22872
I20260620 14:53:10.563486 22934 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 14:53:10.676635 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskUci3DU/build/asan/bin/kudu with pid 22872
I20260620 14:53:10.697711 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskUci3DU/build/asan/bin/kudu pid 22947
I20260620 14:53:10.698959 23009 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 14:53:10.813653 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskUci3DU/build/asan/bin/kudu with pid 22947
2026-06-20T14:53:10Z chronyd exiting
[ OK ] ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys/0 (4942 ms)
[ RUN ] ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys/1
Loading random data
Initializing database '/tmp/dist-test-taskUci3DU/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781966829091300-388-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 20 14:53:10 dist-test-slave-q9j6 krb5kdc[23207](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 20 14:53:10 dist-test-slave-q9j6 krb5kdc[23207](info): set up 2 sockets
Jun 20 14:53:10 dist-test-slave-q9j6 krb5kdc[23207](info): commencing operation
krb5kdc: starting...
W20260620 14:53:11.932299 388 mini_kdc.cc:121] Time spent starting KDC: real 1.077s user 0.000s sys 0.030s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskUci3DU/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781966829091300-388-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskUci3DU/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781966829091300-388-0/krb5kdc/test-user.keytab.
Jun 20 14:53:12 dist-test-slave-q9j6 krb5kdc[23207](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781967192, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-20T14:53:12Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T14:53:12Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.97.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskUci3DU/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781966829091300-388-0/krb5kdc/kudu_127.0.97.62.keytab.
Entry for principal kudu/127.0.97.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskUci3DU/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781966829091300-388-0/krb5kdc/kudu_127.0.97.62.keytab.
I20260620 14:53:12.211374 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskUci3DU/build/asan/bin/kudu
/tmp/dist-test-taskUci3DU/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskUci3DU/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781966829091300-388-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskUci3DU/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781966829091300-388-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskUci3DU/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781966829091300-388-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskUci3DU/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781966829091300-388-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:35275
--webserver_interface=127.0.97.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskUci3DU/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781966829091300-388-0/krb5kdc/kudu_127.0.97.62.keytab
--principal=kudu/127.0.97.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:40241
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.97.62:35275,127.0.97.61:38589,127.0.97.60:38493
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=required with env {KRB5CCNAME=/tmp/dist-test-taskUci3DU/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781966829091300-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskUci3DU/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781966829091300-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskUci3DU/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781966829091300-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 14:53:12.564944 23220 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260620 14:53:12.565366 23220 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 14:53:12.565503 23220 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 14:53:12.565637 23220 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 14:53:12.573307 23220 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 14:53:12.573422 23220 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 14:53:12.573534 23220 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 14:53:12.573621 23220 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 14:53:12.573746 23220 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 14:53:12.584218 23220 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:40241
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-taskUci3DU/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781966829091300-388-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskUci3DU/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781966829091300-388-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.97.62:35275,127.0.97.61:38589,127.0.97.60:38493
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskUci3DU/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781966829091300-388-0/krb5kdc/kudu_127.0.97.62.keytab
--principal=kudu/127.0.97.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:35275
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskUci3DU/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781966829091300-388-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.97.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskUci3DU/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781966829091300-388-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 13:41:20 UTC on ccycloud.quasar-vdagoi.root.comops.site
build id 4457462
ASAN enabled
I20260620 14:53:12.585716 23220 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 14:53:12.587162 23220 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 14:53:12.599928 23225 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 14:53:12.600212 23226 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 14:53:12.600265 23228 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 14:53:12.601094 23220 server_base.cc:1034] running on GCE node
I20260620 14:53:12.601917 23220 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 14:53:12.603418 23220 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 14:53:12.604687 23220 hybrid_clock.cc:648] HybridClock initialized: now 1781967192604608 us; error 83 us; skew 500 ppm
I20260620 14:53:12.605041 23220 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 14:53:12 dist-test-slave-q9j6 krb5kdc[23207](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781967192, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 14:53:12.608484 23220 init.cc:377] Logged in from keytab as kudu/127.0.97.62@KRBTEST.COM (short username kudu)
I20260620 14:53:12.609921 23220 webserver.cc:460] Webserver started at http://127.0.97.62:43757/ using document root <none> and password file <none>
I20260620 14:53:12.610613 23220 fs_manager.cc:359] Metadata directory not provided
I20260620 14:53:12.610698 23220 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 14:53:12.610953 23220 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 14:53:12.614992 23220 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskUci3DU/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781966829091300-388-0/minicluster-data/master-0/data/instance:
uuid: "ad601311ce144f35937a457f9694953f"
format_stamp: "Formatted at 2026-06-20 14:53:12 on dist-test-slave-q9j6"
server_key: "60bd6976de0a5d0667520e70707a9d7a"
server_key_iv: "c821d820119707aa6154600296fde64e"
server_key_version: "encryptionkey@0"
I20260620 14:53:12.616276 23220 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskUci3DU/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781966829091300-388-0/minicluster-data/master-0/wal/instance:
uuid: "ad601311ce144f35937a457f9694953f"
format_stamp: "Formatted at 2026-06-20 14:53:12 on dist-test-slave-q9j6"
server_key: "60bd6976de0a5d0667520e70707a9d7a"
server_key_iv: "c821d820119707aa6154600296fde64e"
server_key_version: "encryptionkey@0"
I20260620 14:53:12.624007 23220 fs_manager.cc:691] Time spent creating directory manager: real 0.007s user 0.006s sys 0.001s
I20260620 14:53:12.627724 23235 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 14:53:12.629309 23220 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260620 14:53:12.629520 23220 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskUci3DU/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781966829091300-388-0/minicluster-data/master-0/data,/tmp/dist-test-taskUci3DU/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781966829091300-388-0/minicluster-data/master-0/wal
uuid: "ad601311ce144f35937a457f9694953f"
format_stamp: "Formatted at 2026-06-20 14:53:12 on dist-test-slave-q9j6"
server_key: "60bd6976de0a5d0667520e70707a9d7a"
server_key_iv: "c821d820119707aa6154600296fde64e"
server_key_version: "encryptionkey@0"
I20260620 14:53:12.629799 23220 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskUci3DU/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781966829091300-388-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskUci3DU/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781966829091300-388-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskUci3DU/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781966829091300-388-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 14:53:12.649420 23220 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 14:53:12.654536 23220 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 14:53:12.654851 23220 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 14:53:12.675719 23220 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.62:35275
I20260620 14:53:12.675782 23286 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.62:35275 every 8 connection(s)
I20260620 14:53:12.677330 23220 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskUci3DU/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781966829091300-388-0/minicluster-data/master-0/data/info.pb
I20260620 14:53:12.679389 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskUci3DU/build/asan/bin/kudu as pid 23220
I20260620 14:53:12.679589 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskUci3DU/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781966829091300-388-0/minicluster-data/master-0/wal/instance
I20260620 14:53:12.680071 388 external_mini_cluster.cc:1428] Setting key 4a97435cf420772c4d78245a5a50b750
I20260620 14:53:12.681859 23287 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 14:53:12.689353 23287 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.97.62" port: 35275 } has no permanent_uuid. Determining permanent_uuid...
WARNING: no policy specified for kudu/127.0.97.61@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.61@KRBTEST.COM" created.
Jun 20 14:53:12 dist-test-slave-q9j6 krb5kdc[23207](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781967192, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.62@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260620 14:53:12.721642 23291 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:53:12.693877 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:35275 (local address 127.0.0.1:58794)
0620 14:53:12.695170 (+ 1293us) negotiation.cc:107] Waiting for socket to connect
0620 14:53:12.695221 (+ 51us) client_negotiation.cc:174] Beginning negotiation
0620 14:53:12.698244 (+ 3023us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 14:53:12.699377 (+ 1133us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 14:53:12.699388 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 14:53:12.700360 (+ 972us) client_negotiation.cc:189] Negotiated authn=SASL
0620 14:53:12.700943 (+ 583us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:53:12.700992 (+ 49us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:53:12.702711 (+ 1719us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:53:12.702720 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:53:12.705021 (+ 2301us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:53:12.705047 (+ 26us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:53:12.706180 (+ 1133us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:53:12.706191 (+ 11us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:53:12.706419 (+ 228us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:53:12.707192 (+ 773us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 14:53:12.707231 (+ 39us) client_negotiation.cc:656] calling sasl_client_start()
0620 14:53:12.712502 (+ 5271us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 14:53:12.716314 (+ 3812us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 14:53:12.716335 (+ 21us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 14:53:12.716354 (+ 19us) client_negotiation.cc:760] calling sasl_client_step()
0620 14:53:12.717087 (+ 733us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 14:53:12.717591 (+ 504us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 14:53:12.717600 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 14:53:12.717606 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0620 14:53:12.717894 (+ 288us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 14:53:12.718930 (+ 1036us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 14:53:12.718944 (+ 14us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 14:53:12.719337 (+ 393us) client_negotiation.cc:769] Sending connection context
0620 14:53:12.719706 (+ 369us) client_negotiation.cc:240] Negotiation successful
0620 14:53:12.720549 (+ 843us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":987,"mutex_wait_us":141,"spinlock_wait_cycles":40704,"thread_start_us":759,"threads_started":1}
I20260620 14:53:12.721686 23290 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:53:12.693878 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58794 (local address 127.0.97.62:35275)
0620 14:53:12.694852 (+ 974us) server_negotiation.cc:206] Beginning negotiation
0620 14:53:12.694880 (+ 28us) server_negotiation.cc:399] Waiting for connection header
0620 14:53:12.695384 (+ 504us) server_negotiation.cc:407] Connection header received
0620 14:53:12.698762 (+ 3378us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:53:12.698769 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:53:12.699084 (+ 315us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:53:12.699258 (+ 174us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:53:12.701230 (+ 1972us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:53:12.702494 (+ 1264us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:53:12.705230 (+ 2736us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:53:12.705861 (+ 631us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:53:12.706030 (+ 169us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:53:12.712842 (+ 6812us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 14:53:12.712852 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 14:53:12.712868 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 14:53:12.712907 (+ 39us) server_negotiation.cc:922] calling sasl_server_start()
0620 14:53:12.716114 (+ 3207us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:53:12.717270 (+ 1156us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:53:12.717277 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:53:12.717280 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:53:12.717444 (+ 164us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:53:12.718019 (+ 575us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:53:12.718022 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:53:12.718024 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:53:12.718702 (+ 678us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 14:53:12.718921 (+ 219us) server_negotiation.cc:1035] Waiting for connection context
0620 14:53:12.719937 (+ 1016us) server_negotiation.cc:299] Negotiation successful
0620 14:53:12.720237 (+ 300us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":677,"thread_start_us":402,"threads_started":1}
I20260620 14:53:12.725322 23287 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.97.61" port: 38589 } has no permanent_uuid. Determining permanent_uuid...
I20260620 14:53:12.726603 23291 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:53:12.726033 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.61:38589 (local address 127.0.0.1:54514)
0620 14:53:12.726259 (+ 226us) negotiation.cc:107] Waiting for socket to connect
0620 14:53:12.726413 (+ 154us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.61:38589: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":73}
W20260620 14:53:12.726910 23239 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.97.61:38589: connect: Connection refused (error 111)
I20260620 14:53:12.728860 23291 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:53:12.728354 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.61:38589 (local address 127.0.0.1:54518)
0620 14:53:12.728561 (+ 207us) negotiation.cc:107] Waiting for socket to connect
0620 14:53:12.728659 (+ 98us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.61:38589: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":57}
W20260620 14:53:12.729105 23287 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.97.61:38589: Network error: Client connection negotiation failed: client connection to 127.0.97.61:38589: connect: Connection refused (error 111)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.61 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskUci3DU/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781966829091300-388-0/krb5kdc/kudu_127.0.97.61.keytab.
Entry for principal kudu/127.0.97.61 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskUci3DU/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781966829091300-388-0/krb5kdc/kudu_127.0.97.61.keytab.
I20260620 14:53:12.746469 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskUci3DU/build/asan/bin/kudu
/tmp/dist-test-taskUci3DU/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskUci3DU/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781966829091300-388-0/minicluster-data/master-1/wal
--fs_data_dirs=/tmp/dist-test-taskUci3DU/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781966829091300-388-0/minicluster-data/master-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskUci3DU/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781966829091300-388-0/minicluster-data/master-1/logs
--server_dump_info_path=/tmp/dist-test-taskUci3DU/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781966829091300-388-0/minicluster-data/master-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.61:38589
--webserver_interface=127.0.97.61
--webserver_port=0
--keytab_file=/tmp/dist-test-taskUci3DU/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781966829091300-388-0/krb5kdc/kudu_127.0.97.61.keytab
--principal=kudu/127.0.97.61
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:40241
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.97.62:35275,127.0.97.61:38589,127.0.97.60:38493
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=required with env {KRB5CCNAME=/tmp/dist-test-taskUci3DU/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781966829091300-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskUci3DU/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781966829091300-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskUci3DU/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781966829091300-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260620 14:53:12.778523 23287 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.97.61" port: 38589 } attempt: 1
I20260620 14:53:12.779889 23291 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:53:12.779293 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.61:38589 (local address 127.0.0.1:54520)
0620 14:53:12.779535 (+ 242us) negotiation.cc:107] Waiting for socket to connect
0620 14:53:12.779700 (+ 165us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.61:38589: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":89}
I20260620 14:53:12.781061 23291 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:53:12.780608 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.61:38589 (local address 127.0.0.1:54536)
0620 14:53:12.780784 (+ 176us) negotiation.cc:107] Waiting for socket to connect
0620 14:53:12.780874 (+ 90us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.61:38589: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":63}
W20260620 14:53:12.781316 23287 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.97.61:38589: Network error: Client connection negotiation failed: client connection to 127.0.97.61:38589: connect: Connection refused (error 111)
I20260620 14:53:12.849632 23287 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.97.61" port: 38589 } attempt: 2
I20260620 14:53:12.851132 23291 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:53:12.850452 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.61:38589 (local address 127.0.0.1:54552)
0620 14:53:12.850785 (+ 333us) negotiation.cc:107] Waiting for socket to connect
0620 14:53:12.850923 (+ 138us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.61:38589: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":99}
I20260620 14:53:12.852314 23291 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:53:12.851878 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.61:38589 (local address 127.0.0.1:54564)
0620 14:53:12.852054 (+ 176us) negotiation.cc:107] Waiting for socket to connect
0620 14:53:12.852145 (+ 91us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.61:38589: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":50}
W20260620 14:53:12.852561 23287 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.97.61:38589: Network error: Client connection negotiation failed: client connection to 127.0.97.61:38589: connect: Connection refused (error 111)
I20260620 14:53:12.943857 23287 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.97.61" port: 38589 } attempt: 3
I20260620 14:53:12.945120 23291 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:53:12.944582 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.61:38589 (local address 127.0.0.1:54568)
0620 14:53:12.944824 (+ 242us) negotiation.cc:107] Waiting for socket to connect
0620 14:53:12.944941 (+ 117us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.61:38589: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":91}
I20260620 14:53:12.946264 23291 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:53:12.945825 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.61:38589 (local address 127.0.0.1:54580)
0620 14:53:12.946009 (+ 184us) negotiation.cc:107] Waiting for socket to connect
0620 14:53:12.946102 (+ 93us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.61:38589: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":63}
W20260620 14:53:12.946523 23287 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.97.61:38589: Network error: Client connection negotiation failed: client connection to 127.0.97.61:38589: connect: Connection refused (error 111)
I20260620 14:53:13.089829 23287 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.97.61" port: 38589 } attempt: 4
I20260620 14:53:13.091223 23291 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:53:13.090585 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.61:38589 (local address 127.0.0.1:54584)
0620 14:53:13.090924 (+ 339us) negotiation.cc:107] Waiting for socket to connect
0620 14:53:13.091037 (+ 113us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.61:38589: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":171}
W20260620 14:53:13.091140 23294 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260620 14:53:13.091662 23294 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 14:53:13.091784 23294 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 14:53:13.091948 23294 flags.cc:425] Enabled unsafe flag: --never_fsync=true
I20260620 14:53:13.092458 23291 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:53:13.092111 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.61:38589 (local address 127.0.0.1:54590)
0620 14:53:13.092242 (+ 131us) negotiation.cc:107] Waiting for socket to connect
0620 14:53:13.092329 (+ 87us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.61:38589: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":37}
W20260620 14:53:13.092690 23287 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.97.61:38589: Network error: Client connection negotiation failed: client connection to 127.0.97.61:38589: connect: Connection refused (error 111)
W20260620 14:53:13.099843 23294 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 14:53:13.099954 23294 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 14:53:13.100059 23294 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 14:53:13.100136 23294 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 14:53:13.100246 23294 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 14:53:13.110772 23294 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:40241
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-taskUci3DU/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781966829091300-388-0/minicluster-data/master-1/data
--fs_wal_dir=/tmp/dist-test-taskUci3DU/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781966829091300-388-0/minicluster-data/master-1/wal
--ipki_ca_key_size=768
--master_addresses=127.0.97.62:35275,127.0.97.61:38589,127.0.97.60:38493
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskUci3DU/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781966829091300-388-0/krb5kdc/kudu_127.0.97.61.keytab
--principal=kudu/127.0.97.61
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.61:38589
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskUci3DU/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781966829091300-388-0/minicluster-data/master-1/data/info.pb
--webserver_interface=127.0.97.61
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskUci3DU/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781966829091300-388-0/minicluster-data/master-1/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 13:41:20 UTC on ccycloud.quasar-vdagoi.root.comops.site
build id 4457462
ASAN enabled
I20260620 14:53:13.112329 23294 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 14:53:13.113755 23294 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 14:53:13.126309 23299 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 14:53:13.126420 23300 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 14:53:13.126390 23302 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 14:53:13.127677 23294 server_base.cc:1034] running on GCE node
I20260620 14:53:13.128585 23294 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 14:53:13.130122 23294 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 14:53:13.131359 23294 hybrid_clock.cc:648] HybridClock initialized: now 1781967193131261 us; error 67 us; skew 500 ppm
I20260620 14:53:13.131856 23294 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 14:53:13 dist-test-slave-q9j6 krb5kdc[23207](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781967193, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.61@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 14:53:13.135314 23294 init.cc:377] Logged in from keytab as kudu/127.0.97.61@KRBTEST.COM (short username kudu)
I20260620 14:53:13.137076 23294 webserver.cc:460] Webserver started at http://127.0.97.61:36133/ using document root <none> and password file <none>
I20260620 14:53:13.137874 23294 fs_manager.cc:359] Metadata directory not provided
I20260620 14:53:13.138019 23294 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 14:53:13.138384 23294 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 14:53:13.141755 23294 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskUci3DU/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781966829091300-388-0/minicluster-data/master-1/data/instance:
uuid: "00c5d93a338047ba8d85935a5cbeab53"
format_stamp: "Formatted at 2026-06-20 14:53:13 on dist-test-slave-q9j6"
server_key: "122671c1dc38dcd69aec23be9dbce2ea"
server_key_iv: "65c836dbc2528020e95ba8f803c0fa5e"
server_key_version: "encryptionkey@0"
I20260620 14:53:13.142683 23294 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskUci3DU/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781966829091300-388-0/minicluster-data/master-1/wal/instance:
uuid: "00c5d93a338047ba8d85935a5cbeab53"
format_stamp: "Formatted at 2026-06-20 14:53:13 on dist-test-slave-q9j6"
server_key: "122671c1dc38dcd69aec23be9dbce2ea"
server_key_iv: "65c836dbc2528020e95ba8f803c0fa5e"
server_key_version: "encryptionkey@0"
I20260620 14:53:13.148767 23294 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.007s sys 0.000s
I20260620 14:53:13.152906 23309 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 14:53:13.154492 23294 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.001s
I20260620 14:53:13.154732 23294 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskUci3DU/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781966829091300-388-0/minicluster-data/master-1/data,/tmp/dist-test-taskUci3DU/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781966829091300-388-0/minicluster-data/master-1/wal
uuid: "00c5d93a338047ba8d85935a5cbeab53"
format_stamp: "Formatted at 2026-06-20 14:53:13 on dist-test-slave-q9j6"
server_key: "122671c1dc38dcd69aec23be9dbce2ea"
server_key_iv: "65c836dbc2528020e95ba8f803c0fa5e"
server_key_version: "encryptionkey@0"
I20260620 14:53:13.155014 23294 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskUci3DU/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781966829091300-388-0/minicluster-data/master-1/wal
metadata directory: /tmp/dist-test-taskUci3DU/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781966829091300-388-0/minicluster-data/master-1/wal
1 data directories: /tmp/dist-test-taskUci3DU/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781966829091300-388-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 14:53:13.189066 23294 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 14:53:13.193456 23294 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 14:53:13.193784 23294 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 14:53:13.212939 23294 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.61:38589
I20260620 14:53:13.212960 23360 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.61:38589 every 8 connection(s)
I20260620 14:53:13.214419 23294 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskUci3DU/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781966829091300-388-0/minicluster-data/master-1/data/info.pb
I20260620 14:53:13.218479 23361 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 14:53:13.224155 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskUci3DU/build/asan/bin/kudu as pid 23294
I20260620 14:53:13.224371 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskUci3DU/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781966829091300-388-0/minicluster-data/master-1/wal/instance
I20260620 14:53:13.224905 388 external_mini_cluster.cc:1428] Setting key 380c5bebf612f6fcb0c60994b796c8c0
I20260620 14:53:13.226640 23361 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.97.62" port: 35275 } has no permanent_uuid. Determining permanent_uuid...
Jun 20 14:53:13 dist-test-slave-q9j6 krb5kdc[23207](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781967193, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.61@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
WARNING: no policy specified for kudu/127.0.97.60@KRBTEST.COM; defaulting to no policy
I20260620 14:53:13.250512 23363 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:53:13.230370 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58810 (local address 127.0.97.62:35275)
0620 14:53:13.231317 (+ 947us) server_negotiation.cc:206] Beginning negotiation
0620 14:53:13.231325 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0620 14:53:13.232243 (+ 918us) server_negotiation.cc:407] Connection header received
0620 14:53:13.234511 (+ 2268us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:53:13.234521 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:53:13.234634 (+ 113us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:53:13.234775 (+ 141us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:53:13.236731 (+ 1956us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:53:13.237507 (+ 776us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:53:13.239172 (+ 1665us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:53:13.239677 (+ 505us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:53:13.239815 (+ 138us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:53:13.243585 (+ 3770us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 14:53:13.243591 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 14:53:13.243653 (+ 62us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 14:53:13.243713 (+ 60us) server_negotiation.cc:922] calling sasl_server_start()
0620 14:53:13.246997 (+ 3284us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:53:13.247993 (+ 996us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:53:13.247999 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:53:13.248003 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:53:13.248148 (+ 145us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:53:13.248628 (+ 480us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:53:13.248633 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:53:13.248636 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:53:13.248943 (+ 307us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 14:53:13.249119 (+ 176us) server_negotiation.cc:1035] Waiting for connection context
0620 14:53:13.249886 (+ 767us) server_negotiation.cc:299] Negotiation successful
0620 14:53:13.250122 (+ 236us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":808,"thread_start_us":659,"threads_started":1}
I20260620 14:53:13.251521 23364 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:53:13.231106 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:35275 (local address 127.0.0.1:58810)
0620 14:53:13.232050 (+ 944us) negotiation.cc:107] Waiting for socket to connect
0620 14:53:13.232101 (+ 51us) client_negotiation.cc:174] Beginning negotiation
0620 14:53:13.234140 (+ 2039us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 14:53:13.235109 (+ 969us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 14:53:13.235118 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 14:53:13.235813 (+ 695us) client_negotiation.cc:189] Negotiated authn=SASL
0620 14:53:13.236468 (+ 655us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:53:13.236489 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:53:13.237709 (+ 1220us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:53:13.237713 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:53:13.238990 (+ 1277us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:53:13.239004 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:53:13.239918 (+ 914us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:53:13.239925 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:53:13.240188 (+ 263us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:53:13.240966 (+ 778us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 14:53:13.240991 (+ 25us) client_negotiation.cc:656] calling sasl_client_start()
0620 14:53:13.243290 (+ 2299us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 14:53:13.247222 (+ 3932us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 14:53:13.247234 (+ 12us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 14:53:13.247260 (+ 26us) client_negotiation.cc:760] calling sasl_client_step()
0620 14:53:13.247853 (+ 593us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 14:53:13.248273 (+ 420us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 14:53:13.248278 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 14:53:13.248282 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0620 14:53:13.248493 (+ 211us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 14:53:13.249083 (+ 590us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 14:53:13.249092 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 14:53:13.249496 (+ 404us) client_negotiation.cc:769] Sending connection context
0620 14:53:13.249779 (+ 283us) client_negotiation.cc:240] Negotiation successful
0620 14:53:13.250364 (+ 585us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":644,"thread_start_us":420,"threads_started":1}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.60@KRBTEST.COM" created.
I20260620 14:53:13.253607 23361 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.97.61" port: 38589 } has no permanent_uuid. Determining permanent_uuid...
Jun 20 14:53:13 dist-test-slave-q9j6 krb5kdc[23207](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781967193, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.61@KRBTEST.COM for kudu/127.0.97.61@KRBTEST.COM
I20260620 14:53:13.271404 23364 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:53:13.254323 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.61:38589 (local address 127.0.0.1:54600)
0620 14:53:13.254537 (+ 214us) negotiation.cc:107] Waiting for socket to connect
0620 14:53:13.254558 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0620 14:53:13.254673 (+ 115us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 14:53:13.256754 (+ 2081us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 14:53:13.256757 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 14:53:13.257166 (+ 409us) client_negotiation.cc:189] Negotiated authn=SASL
0620 14:53:13.257337 (+ 171us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:53:13.257346 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:53:13.259384 (+ 2038us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:53:13.259388 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:53:13.260542 (+ 1154us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:53:13.260556 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:53:13.261885 (+ 1329us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:53:13.261889 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:53:13.262020 (+ 131us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:53:13.262682 (+ 662us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 14:53:13.262701 (+ 19us) client_negotiation.cc:656] calling sasl_client_start()
0620 14:53:13.265284 (+ 2583us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 14:53:13.268804 (+ 3520us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 14:53:13.268812 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 14:53:13.268817 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0620 14:53:13.269204 (+ 387us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 14:53:13.269591 (+ 387us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 14:53:13.269597 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 14:53:13.269601 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0620 14:53:13.269743 (+ 142us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 14:53:13.270354 (+ 611us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 14:53:13.270360 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 14:53:13.270522 (+ 162us) client_negotiation.cc:769] Sending connection context
0620 14:53:13.270673 (+ 151us) client_negotiation.cc:240] Negotiation successful
0620 14:53:13.270994 (+ 321us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":65}
I20260620 14:53:13.271518 23366 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:53:13.254638 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54600 (local address 127.0.97.61:38589)
0620 14:53:13.256167 (+ 1529us) server_negotiation.cc:206] Beginning negotiation
0620 14:53:13.256178 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0620 14:53:13.256215 (+ 37us) server_negotiation.cc:407] Connection header received
0620 14:53:13.256279 (+ 64us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:53:13.256289 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:53:13.256567 (+ 278us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:53:13.256738 (+ 171us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:53:13.257522 (+ 784us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:53:13.259224 (+ 1702us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:53:13.260690 (+ 1466us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:53:13.261705 (+ 1015us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:53:13.261872 (+ 167us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:53:13.265523 (+ 3651us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 14:53:13.265536 (+ 13us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 14:53:13.265552 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 14:53:13.265582 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0620 14:53:13.268636 (+ 3054us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:53:13.269348 (+ 712us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:53:13.269367 (+ 19us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:53:13.269370 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:53:13.269480 (+ 110us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:53:13.269854 (+ 374us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:53:13.269859 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:53:13.269862 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:53:13.270227 (+ 365us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 14:53:13.270471 (+ 244us) server_negotiation.cc:1035] Waiting for connection context
0620 14:53:13.270829 (+ 358us) server_negotiation.cc:299] Negotiation successful
0620 14:53:13.271111 (+ 282us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":45,"server-negotiator.queue_time_us":1319,"thread_start_us":1150,"threads_started":1}
I20260620 14:53:13.274132 23361 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.97.60" port: 38493 } has no permanent_uuid. Determining permanent_uuid...
I20260620 14:53:13.275527 23364 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:53:13.275010 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.60:38493 (local address 127.0.0.1:59154)
0620 14:53:13.275216 (+ 206us) negotiation.cc:107] Waiting for socket to connect
0620 14:53:13.275343 (+ 127us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.60:38493: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":68}
W20260620 14:53:13.275921 23310 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.97.60:38493: connect: Connection refused (error 111)
I20260620 14:53:13.277736 23364 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:53:13.277344 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.60:38493 (local address 127.0.0.1:59160)
0620 14:53:13.277527 (+ 183us) negotiation.cc:107] Waiting for socket to connect
0620 14:53:13.277619 (+ 92us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.60:38493: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":70}
W20260620 14:53:13.277977 23361 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.97.60:38493: Network error: Client connection negotiation failed: client connection to 127.0.97.60:38493: connect: Connection refused (error 111)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.60 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskUci3DU/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781966829091300-388-0/krb5kdc/kudu_127.0.97.60.keytab.
Entry for principal kudu/127.0.97.60 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskUci3DU/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781966829091300-388-0/krb5kdc/kudu_127.0.97.60.keytab.
I20260620 14:53:13.288918 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskUci3DU/build/asan/bin/kudu
/tmp/dist-test-taskUci3DU/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskUci3DU/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781966829091300-388-0/minicluster-data/master-2/wal
--fs_data_dirs=/tmp/dist-test-taskUci3DU/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781966829091300-388-0/minicluster-data/master-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskUci3DU/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781966829091300-388-0/minicluster-data/master-2/logs
--server_dump_info_path=/tmp/dist-test-taskUci3DU/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781966829091300-388-0/minicluster-data/master-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.60:38493
--webserver_interface=127.0.97.60
--webserver_port=0
--keytab_file=/tmp/dist-test-taskUci3DU/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781966829091300-388-0/krb5kdc/kudu_127.0.97.60.keytab
--principal=kudu/127.0.97.60
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:40241
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.97.62:35275,127.0.97.61:38589,127.0.97.60:38493
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=required with env {KRB5CCNAME=/tmp/dist-test-taskUci3DU/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781966829091300-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskUci3DU/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781966829091300-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskUci3DU/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781966829091300-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260620 14:53:13.327282 23361 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.97.60" port: 38493 } attempt: 1
I20260620 14:53:13.328513 23364 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:53:13.328041 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.60:38493 (local address 127.0.0.1:59162)
0620 14:53:13.328239 (+ 198us) negotiation.cc:107] Waiting for socket to connect
0620 14:53:13.328358 (+ 119us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.60:38493: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":69}
I20260620 14:53:13.329612 23364 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:53:13.329243 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.60:38493 (local address 127.0.0.1:59174)
0620 14:53:13.329418 (+ 175us) negotiation.cc:107] Waiting for socket to connect
0620 14:53:13.329502 (+ 84us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.60:38493: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":51}
W20260620 14:53:13.329816 23361 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.97.60:38493: Network error: Client connection negotiation failed: client connection to 127.0.97.60:38493: connect: Connection refused (error 111)
I20260620 14:53:13.392015 23287 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.97.61" port: 38589 } attempt: 5
I20260620 14:53:13.398049 23361 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.97.60" port: 38493 } attempt: 2
I20260620 14:53:13.399210 23364 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:53:13.398695 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.60:38493 (local address 127.0.0.1:59182)
0620 14:53:13.398914 (+ 219us) negotiation.cc:107] Waiting for socket to connect
0620 14:53:13.399025 (+ 111us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.60:38493: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":82}
I20260620 14:53:13.400517 23364 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:53:13.400011 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.60:38493 (local address 127.0.0.1:59188)
0620 14:53:13.400191 (+ 180us) negotiation.cc:107] Waiting for socket to connect
0620 14:53:13.400288 (+ 97us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.60:38493: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":61}
Jun 20 14:53:13 dist-test-slave-q9j6 krb5kdc[23207](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781967192, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.62@KRBTEST.COM for kudu/127.0.97.61@KRBTEST.COM
W20260620 14:53:13.400744 23361 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.97.60:38493: Network error: Client connection negotiation failed: client connection to 127.0.97.60:38493: connect: Connection refused (error 111)
I20260620 14:53:13.407982 23291 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:53:13.392908 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.61:38589 (local address 127.0.0.1:54606)
0620 14:53:13.393155 (+ 247us) negotiation.cc:107] Waiting for socket to connect
0620 14:53:13.393178 (+ 23us) client_negotiation.cc:174] Beginning negotiation
0620 14:53:13.393323 (+ 145us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 14:53:13.393769 (+ 446us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 14:53:13.393772 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 14:53:13.394164 (+ 392us) client_negotiation.cc:189] Negotiated authn=SASL
0620 14:53:13.394349 (+ 185us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:53:13.394360 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:53:13.395799 (+ 1439us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:53:13.395803 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:53:13.396714 (+ 911us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:53:13.396728 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:53:13.397707 (+ 979us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:53:13.397711 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:53:13.397837 (+ 126us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:53:13.398610 (+ 773us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 14:53:13.398633 (+ 23us) client_negotiation.cc:656] calling sasl_client_start()
0620 14:53:13.401284 (+ 2651us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 14:53:13.405337 (+ 4053us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 14:53:13.405342 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 14:53:13.405345 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0620 14:53:13.405798 (+ 453us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 14:53:13.406266 (+ 468us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 14:53:13.406270 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 14:53:13.406272 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 14:53:13.406437 (+ 165us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 14:53:13.407083 (+ 646us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 14:53:13.407087 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 14:53:13.407226 (+ 139us) client_negotiation.cc:769] Sending connection context
0620 14:53:13.407337 (+ 111us) client_negotiation.cc:240] Negotiation successful
0620 14:53:13.407713 (+ 376us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":86}
I20260620 14:53:13.408294 23366 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:53:13.393129 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54606 (local address 127.0.97.61:38589)
0620 14:53:13.393353 (+ 224us) server_negotiation.cc:206] Beginning negotiation
0620 14:53:13.393361 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0620 14:53:13.393416 (+ 55us) server_negotiation.cc:407] Connection header received
0620 14:53:13.393510 (+ 94us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:53:13.393518 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:53:13.393608 (+ 90us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:53:13.393716 (+ 108us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:53:13.394522 (+ 806us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:53:13.395590 (+ 1068us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:53:13.396897 (+ 1307us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:53:13.397551 (+ 654us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:53:13.397683 (+ 132us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:53:13.401505 (+ 3822us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 14:53:13.401512 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 14:53:13.401516 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 14:53:13.401558 (+ 42us) server_negotiation.cc:922] calling sasl_server_start()
0620 14:53:13.405112 (+ 3554us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:53:13.405957 (+ 845us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:53:13.405962 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:53:13.405966 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:53:13.406137 (+ 171us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:53:13.406550 (+ 413us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:53:13.406556 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:53:13.406558 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:53:13.406944 (+ 386us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 14:53:13.407133 (+ 189us) server_negotiation.cc:1035] Waiting for connection context
0620 14:53:13.407505 (+ 372us) server_negotiation.cc:299] Negotiation successful
0620 14:53:13.407841 (+ 336us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":61}
I20260620 14:53:13.409353 23287 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.97.60" port: 38493 } has no permanent_uuid. Determining permanent_uuid...
I20260620 14:53:13.410521 23291 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:53:13.410097 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.60:38493 (local address 127.0.0.1:59196)
0620 14:53:13.410278 (+ 181us) negotiation.cc:107] Waiting for socket to connect
0620 14:53:13.410360 (+ 82us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.60:38493: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":68}
I20260620 14:53:13.411581 23291 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:53:13.411247 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.60:38493 (local address 127.0.0.1:59210)
0620 14:53:13.411393 (+ 146us) negotiation.cc:107] Waiting for socket to connect
0620 14:53:13.411479 (+ 86us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.60:38493: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":52}
W20260620 14:53:13.411911 23287 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.97.60:38493: Network error: Client connection negotiation failed: client connection to 127.0.97.60:38493: connect: Connection refused (error 111)
I20260620 14:53:13.463135 23287 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.97.60" port: 38493 } attempt: 1
I20260620 14:53:13.464514 23291 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:53:13.463985 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.60:38493 (local address 127.0.0.1:59222)
0620 14:53:13.464230 (+ 245us) negotiation.cc:107] Waiting for socket to connect
0620 14:53:13.464337 (+ 107us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.60:38493: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":109}
I20260620 14:53:13.465734 23291 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:53:13.465337 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.60:38493 (local address 127.0.0.1:59228)
0620 14:53:13.465530 (+ 193us) negotiation.cc:107] Waiting for socket to connect
0620 14:53:13.465614 (+ 84us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.60:38493: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":84}
W20260620 14:53:13.465977 23287 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.97.60:38493: Network error: Client connection negotiation failed: client connection to 127.0.97.60:38493: connect: Connection refused (error 111)
I20260620 14:53:13.492026 23361 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.97.60" port: 38493 } attempt: 3
I20260620 14:53:13.493299 23364 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:53:13.492791 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.60:38493 (local address 127.0.0.1:59234)
0620 14:53:13.493046 (+ 255us) negotiation.cc:107] Waiting for socket to connect
0620 14:53:13.493135 (+ 89us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.60:38493: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":91}
I20260620 14:53:13.494478 23364 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:53:13.494025 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.60:38493 (local address 127.0.0.1:59240)
0620 14:53:13.494275 (+ 250us) negotiation.cc:107] Waiting for socket to connect
0620 14:53:13.494354 (+ 79us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.60:38493: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":136}
W20260620 14:53:13.494693 23361 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.97.60:38493: Network error: Client connection negotiation failed: client connection to 127.0.97.60:38493: connect: Connection refused (error 111)
I20260620 14:53:13.534322 23287 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.97.60" port: 38493 } attempt: 2
I20260620 14:53:13.535569 23291 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:53:13.535048 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.60:38493 (local address 127.0.0.1:59250)
0620 14:53:13.535300 (+ 252us) negotiation.cc:107] Waiting for socket to connect
0620 14:53:13.535405 (+ 105us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.60:38493: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":73}
I20260620 14:53:13.536744 23291 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:53:13.536392 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.60:38493 (local address 127.0.0.1:59264)
0620 14:53:13.536546 (+ 154us) negotiation.cc:107] Waiting for socket to connect
0620 14:53:13.536622 (+ 76us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.60:38493: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":56}
W20260620 14:53:13.536996 23287 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.97.60:38493: Network error: Client connection negotiation failed: client connection to 127.0.97.60:38493: connect: Connection refused (error 111)
I20260620 14:53:13.637936 23361 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.97.60" port: 38493 } attempt: 4
I20260620 14:53:13.639050 23364 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:53:13.638575 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.60:38493 (local address 127.0.0.1:59274)
0620 14:53:13.638811 (+ 236us) negotiation.cc:107] Waiting for socket to connect
0620 14:53:13.638897 (+ 86us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.60:38493: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":67}
I20260620 14:53:13.640189 23364 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:53:13.639850 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.60:38493 (local address 127.0.0.1:59288)
0620 14:53:13.640007 (+ 157us) negotiation.cc:107] Waiting for socket to connect
0620 14:53:13.640081 (+ 74us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.60:38493: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":51}
W20260620 14:53:13.640451 23361 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.97.60:38493: Network error: Client connection negotiation failed: client connection to 127.0.97.60:38493: connect: Connection refused (error 111)
W20260620 14:53:13.642899 23369 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260620 14:53:13.643275 23369 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
I20260620 14:53:13.643234 23287 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.97.60" port: 38493 } attempt: 3
W20260620 14:53:13.643378 23369 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 14:53:13.643539 23369 flags.cc:425] Enabled unsafe flag: --never_fsync=true
I20260620 14:53:13.644369 23291 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:53:13.643941 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.60:38493 (local address 127.0.0.1:59290)
0620 14:53:13.644135 (+ 194us) negotiation.cc:107] Waiting for socket to connect
0620 14:53:13.644219 (+ 84us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.60:38493: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":77}
I20260620 14:53:13.645402 23291 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:53:13.644994 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.60:38493 (local address 127.0.0.1:59304)
0620 14:53:13.645177 (+ 183us) negotiation.cc:107] Waiting for socket to connect
0620 14:53:13.645247 (+ 70us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.60:38493: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":87}
W20260620 14:53:13.645635 23287 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.97.60:38493: Network error: Client connection negotiation failed: client connection to 127.0.97.60:38493: connect: Connection refused (error 111)
W20260620 14:53:13.651965 23369 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 14:53:13.652074 23369 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 14:53:13.652172 23369 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 14:53:13.652236 23369 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 14:53:13.652364 23369 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 14:53:13.662969 23369 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:40241
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-taskUci3DU/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781966829091300-388-0/minicluster-data/master-2/data
--fs_wal_dir=/tmp/dist-test-taskUci3DU/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781966829091300-388-0/minicluster-data/master-2/wal
--ipki_ca_key_size=768
--master_addresses=127.0.97.62:35275,127.0.97.61:38589,127.0.97.60:38493
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskUci3DU/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781966829091300-388-0/krb5kdc/kudu_127.0.97.60.keytab
--principal=kudu/127.0.97.60
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.60:38493
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskUci3DU/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781966829091300-388-0/minicluster-data/master-2/data/info.pb
--webserver_interface=127.0.97.60
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskUci3DU/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781966829091300-388-0/minicluster-data/master-2/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 13:41:20 UTC on ccycloud.quasar-vdagoi.root.comops.site
build id 4457462
ASAN enabled
I20260620 14:53:13.664564 23369 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 14:53:13.666025 23369 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 14:53:13.679052 23374 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 14:53:13.678850 23375 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 14:53:13.678894 23377 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 14:53:13.680017 23369 server_base.cc:1034] running on GCE node
I20260620 14:53:13.680920 23369 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 14:53:13.682565 23369 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 14:53:13.683844 23369 hybrid_clock.cc:648] HybridClock initialized: now 1781967193683761 us; error 64 us; skew 500 ppm
I20260620 14:53:13.684223 23369 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 14:53:13 dist-test-slave-q9j6 krb5kdc[23207](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781967193, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.60@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 14:53:13.687795 23369 init.cc:377] Logged in from keytab as kudu/127.0.97.60@KRBTEST.COM (short username kudu)
I20260620 14:53:13.689360 23369 webserver.cc:460] Webserver started at http://127.0.97.60:34955/ using document root <none> and password file <none>
I20260620 14:53:13.690095 23369 fs_manager.cc:359] Metadata directory not provided
I20260620 14:53:13.690187 23369 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 14:53:13.690478 23369 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 14:53:13.693696 23369 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskUci3DU/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781966829091300-388-0/minicluster-data/master-2/data/instance:
uuid: "53df900c7e424f3e9824e63bd97f4bf7"
format_stamp: "Formatted at 2026-06-20 14:53:13 on dist-test-slave-q9j6"
server_key: "0a381a7b7c7f60eb798ffce6f2f9a3c8"
server_key_iv: "6813f2c9491923f063275ab4896c68bb"
server_key_version: "encryptionkey@0"
I20260620 14:53:13.694515 23369 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskUci3DU/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781966829091300-388-0/minicluster-data/master-2/wal/instance:
uuid: "53df900c7e424f3e9824e63bd97f4bf7"
format_stamp: "Formatted at 2026-06-20 14:53:13 on dist-test-slave-q9j6"
server_key: "0a381a7b7c7f60eb798ffce6f2f9a3c8"
server_key_iv: "6813f2c9491923f063275ab4896c68bb"
server_key_version: "encryptionkey@0"
I20260620 14:53:13.700372 23369 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.003s sys 0.004s
I20260620 14:53:13.704241 23384 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 14:53:13.705838 23369 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260620 14:53:13.706066 23369 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskUci3DU/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781966829091300-388-0/minicluster-data/master-2/data,/tmp/dist-test-taskUci3DU/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781966829091300-388-0/minicluster-data/master-2/wal
uuid: "53df900c7e424f3e9824e63bd97f4bf7"
format_stamp: "Formatted at 2026-06-20 14:53:13 on dist-test-slave-q9j6"
server_key: "0a381a7b7c7f60eb798ffce6f2f9a3c8"
server_key_iv: "6813f2c9491923f063275ab4896c68bb"
server_key_version: "encryptionkey@0"
I20260620 14:53:13.706342 23369 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskUci3DU/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781966829091300-388-0/minicluster-data/master-2/wal
metadata directory: /tmp/dist-test-taskUci3DU/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781966829091300-388-0/minicluster-data/master-2/wal
1 data directories: /tmp/dist-test-taskUci3DU/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781966829091300-388-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 14:53:13.725540 23369 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 14:53:13.730516 23369 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 14:53:13.730827 23369 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 14:53:13.750365 23369 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.60:38493
I20260620 14:53:13.750409 23435 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.60:38493 every 8 connection(s)
I20260620 14:53:13.752043 23369 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskUci3DU/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781966829091300-388-0/minicluster-data/master-2/data/info.pb
I20260620 14:53:13.756325 23436 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 14:53:13.757692 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskUci3DU/build/asan/bin/kudu as pid 23369
I20260620 14:53:13.757881 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskUci3DU/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781966829091300-388-0/minicluster-data/master-2/wal/instance
I20260620 14:53:13.758317 388 external_mini_cluster.cc:1428] Setting key 2012305156554ac153a5d6ccd8d389e2
I20260620 14:53:13.766191 23436 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.97.62" port: 35275 } has no permanent_uuid. Determining permanent_uuid...
Jun 20 14:53:13 dist-test-slave-q9j6 krb5kdc[23207](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781967192, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.60@KRBTEST.COM
Jun 20 14:53:13 dist-test-slave-q9j6 krb5kdc[23207](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781967193, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.60@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260620 14:53:13.784948 23440 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:53:13.769426 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58816 (local address 127.0.97.62:35275)
0620 14:53:13.770013 (+ 587us) server_negotiation.cc:206] Beginning negotiation
0620 14:53:13.770020 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0620 14:53:13.770570 (+ 550us) server_negotiation.cc:407] Connection header received
0620 14:53:13.770755 (+ 185us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:53:13.770760 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:53:13.770841 (+ 81us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:53:13.770979 (+ 138us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:53:13.772220 (+ 1241us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:53:13.772931 (+ 711us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:53:13.774307 (+ 1376us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:53:13.774723 (+ 416us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:53:13.774865 (+ 142us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:53:13.778615 (+ 3750us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 14:53:13.778619 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 14:53:13.778623 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 14:53:13.778657 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0620 14:53:13.781808 (+ 3151us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:53:13.782721 (+ 913us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:53:13.782725 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:53:13.782727 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:53:13.782858 (+ 131us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:53:13.783221 (+ 363us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:53:13.783225 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:53:13.783227 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:53:13.783485 (+ 258us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 14:53:13.783678 (+ 193us) server_negotiation.cc:1035] Waiting for connection context
0620 14:53:13.784450 (+ 772us) server_negotiation.cc:299] Negotiation successful
0620 14:53:13.784664 (+ 214us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":423,"thread_start_us":266,"threads_started":1}
I20260620 14:53:13.786260 23441 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:53:13.769481 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:35275 (local address 127.0.0.1:58816)
0620 14:53:13.770369 (+ 888us) negotiation.cc:107] Waiting for socket to connect
0620 14:53:13.770417 (+ 48us) client_negotiation.cc:174] Beginning negotiation
0620 14:53:13.770602 (+ 185us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 14:53:13.771045 (+ 443us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 14:53:13.771052 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 14:53:13.771796 (+ 744us) client_negotiation.cc:189] Negotiated authn=SASL
0620 14:53:13.772048 (+ 252us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:53:13.772060 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:53:13.773106 (+ 1046us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:53:13.773116 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:53:13.774141 (+ 1025us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:53:13.774153 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:53:13.774966 (+ 813us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:53:13.774973 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:53:13.775136 (+ 163us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:53:13.775836 (+ 700us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 14:53:13.775861 (+ 25us) client_negotiation.cc:656] calling sasl_client_start()
0620 14:53:13.778378 (+ 2517us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 14:53:13.781955 (+ 3577us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 14:53:13.781966 (+ 11us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 14:53:13.781977 (+ 11us) client_negotiation.cc:760] calling sasl_client_step()
0620 14:53:13.782589 (+ 612us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 14:53:13.782971 (+ 382us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 14:53:13.782976 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 14:53:13.782980 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0620 14:53:13.783118 (+ 138us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 14:53:13.783689 (+ 571us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 14:53:13.783697 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 14:53:13.783962 (+ 265us) client_negotiation.cc:769] Sending connection context
0620 14:53:13.784298 (+ 336us) client_negotiation.cc:240] Negotiation successful
0620 14:53:13.784849 (+ 551us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":621,"mutex_wait_us":202,"spinlock_wait_cycles":22272,"thread_start_us":456,"threads_started":1}
I20260620 14:53:13.786279 23439 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:53:13.760294 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59310 (local address 127.0.97.60:38493)
0620 14:53:13.761148 (+ 854us) server_negotiation.cc:206] Beginning negotiation
0620 14:53:13.761161 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0620 14:53:13.761207 (+ 46us) server_negotiation.cc:407] Connection header received
0620 14:53:13.762847 (+ 1640us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:53:13.762874 (+ 27us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:53:13.763345 (+ 471us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:53:13.763877 (+ 532us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:53:13.765653 (+ 1776us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:53:13.766970 (+ 1317us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:53:13.768872 (+ 1902us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:53:13.770137 (+ 1265us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:53:13.770327 (+ 190us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:53:13.778190 (+ 7863us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 14:53:13.778205 (+ 15us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 14:53:13.778225 (+ 20us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 14:53:13.778269 (+ 44us) server_negotiation.cc:922] calling sasl_server_start()
0620 14:53:13.781180 (+ 2911us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:53:13.782137 (+ 957us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:53:13.782147 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:53:13.782151 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:53:13.782338 (+ 187us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:53:13.782706 (+ 368us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:53:13.782713 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:53:13.782718 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:53:13.783446 (+ 728us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 14:53:13.783781 (+ 335us) server_negotiation.cc:1035] Waiting for connection context
0620 14:53:13.784172 (+ 391us) server_negotiation.cc:299] Negotiation successful
0620 14:53:13.784727 (+ 555us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":42,"server-negotiator.queue_time_us":607,"thread_start_us":385,"threads_started":1}
I20260620 14:53:13.788038 23436 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.97.61" port: 38589 } has no permanent_uuid. Determining permanent_uuid...
Jun 20 14:53:13 dist-test-slave-q9j6 krb5kdc[23207](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781967193, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.60@KRBTEST.COM for kudu/127.0.97.61@KRBTEST.COM
Jun 20 14:53:13 dist-test-slave-q9j6 krb5kdc[23207](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781967192, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260620 14:53:13.802866 23441 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:53:13.788685 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.61:38589 (local address 127.0.0.1:54614)
0620 14:53:13.788887 (+ 202us) negotiation.cc:107] Waiting for socket to connect
0620 14:53:13.788909 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0620 14:53:13.789008 (+ 99us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 14:53:13.789428 (+ 420us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 14:53:13.789436 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 14:53:13.789910 (+ 474us) client_negotiation.cc:189] Negotiated authn=SASL
0620 14:53:13.790071 (+ 161us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:53:13.790080 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:53:13.791134 (+ 1054us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:53:13.791142 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:53:13.792229 (+ 1087us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:53:13.792242 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:53:13.793004 (+ 762us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:53:13.793010 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:53:13.793204 (+ 194us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:53:13.793826 (+ 622us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 14:53:13.793856 (+ 30us) client_negotiation.cc:656] calling sasl_client_start()
0620 14:53:13.796302 (+ 2446us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 14:53:13.800108 (+ 3806us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 14:53:13.800113 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 14:53:13.800116 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0620 14:53:13.800535 (+ 419us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 14:53:13.801252 (+ 717us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 14:53:13.801256 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 14:53:13.801258 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 14:53:13.801394 (+ 136us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 14:53:13.802006 (+ 612us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 14:53:13.802009 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 14:53:13.802143 (+ 134us) client_negotiation.cc:769] Sending connection context
0620 14:53:13.802263 (+ 120us) client_negotiation.cc:240] Negotiation successful
0620 14:53:13.802477 (+ 214us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":65}
I20260620 14:53:13.802911 23366 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:53:13.788873 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54614 (local address 127.0.97.61:38589)
0620 14:53:13.789087 (+ 214us) server_negotiation.cc:206] Beginning negotiation
0620 14:53:13.789093 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 14:53:13.789113 (+ 20us) server_negotiation.cc:407] Connection header received
0620 14:53:13.789176 (+ 63us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:53:13.789182 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:53:13.789259 (+ 77us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:53:13.789374 (+ 115us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:53:13.790199 (+ 825us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:53:13.790970 (+ 771us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:53:13.792376 (+ 1406us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:53:13.792838 (+ 462us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:53:13.792936 (+ 98us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:53:13.796527 (+ 3591us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 14:53:13.796532 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 14:53:13.796536 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 14:53:13.796571 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0620 14:53:13.799955 (+ 3384us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:53:13.800941 (+ 986us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:53:13.800953 (+ 12us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:53:13.800956 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:53:13.801130 (+ 174us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:53:13.801508 (+ 378us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:53:13.801515 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:53:13.801520 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:53:13.801884 (+ 364us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 14:53:13.802061 (+ 177us) server_negotiation.cc:1035] Waiting for connection context
0620 14:53:13.802403 (+ 342us) server_negotiation.cc:299] Negotiation successful
0620 14:53:13.802651 (+ 248us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":93}
I20260620 14:53:13.804075 23436 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.97.60" port: 38493 } has no permanent_uuid. Determining permanent_uuid...
I20260620 14:53:13.805510 23440 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:53:13.789719 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58820 (local address 127.0.97.62:35275)
0620 14:53:13.789878 (+ 159us) server_negotiation.cc:206] Beginning negotiation
0620 14:53:13.789885 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0620 14:53:13.789906 (+ 21us) server_negotiation.cc:407] Connection header received
0620 14:53:13.790125 (+ 219us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:53:13.790132 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:53:13.790219 (+ 87us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:53:13.790318 (+ 99us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:53:13.791578 (+ 1260us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:53:13.792330 (+ 752us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:53:13.793828 (+ 1498us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:53:13.794263 (+ 435us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:53:13.794359 (+ 96us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:53:13.799013 (+ 4654us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 14:53:13.799020 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 14:53:13.799026 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 14:53:13.799075 (+ 49us) server_negotiation.cc:922] calling sasl_server_start()
0620 14:53:13.802434 (+ 3359us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:53:13.803266 (+ 832us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:53:13.803275 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:53:13.803279 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:53:13.803415 (+ 136us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:53:13.803927 (+ 512us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:53:13.803932 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:53:13.803935 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:53:13.804314 (+ 379us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 14:53:13.804557 (+ 243us) server_negotiation.cc:1035] Waiting for connection context
0620 14:53:13.804916 (+ 359us) server_negotiation.cc:299] Negotiation successful
0620 14:53:13.805172 (+ 256us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":59}
Jun 20 14:53:13 dist-test-slave-q9j6 krb5kdc[23207](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781967193, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.60@KRBTEST.COM for kudu/127.0.97.60@KRBTEST.COM
Jun 20 14:53:13 dist-test-slave-q9j6 krb5kdc[23207](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781967192, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.61@KRBTEST.COM
I20260620 14:53:13.818397 23439 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:53:13.804889 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59314 (local address 127.0.97.60:38493)
0620 14:53:13.805061 (+ 172us) server_negotiation.cc:206] Beginning negotiation
0620 14:53:13.805068 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0620 14:53:13.805234 (+ 166us) server_negotiation.cc:407] Connection header received
0620 14:53:13.805408 (+ 174us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:53:13.805413 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:53:13.805485 (+ 72us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:53:13.805557 (+ 72us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:53:13.806782 (+ 1225us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:53:13.807879 (+ 1097us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:53:13.809040 (+ 1161us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:53:13.809425 (+ 385us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:53:13.809529 (+ 104us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:53:13.813073 (+ 3544us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 14:53:13.813078 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 14:53:13.813081 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 14:53:13.813108 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0620 14:53:13.815576 (+ 2468us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:53:13.816236 (+ 660us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:53:13.816244 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:53:13.816247 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:53:13.816402 (+ 155us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:53:13.816791 (+ 389us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:53:13.816796 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:53:13.816799 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:53:13.817191 (+ 392us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 14:53:13.817357 (+ 166us) server_negotiation.cc:1035] Waiting for connection context
0620 14:53:13.817697 (+ 340us) server_negotiation.cc:299] Negotiation successful
0620 14:53:13.818110 (+ 413us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":182,"server-negotiator.queue_time_us":58}
I20260620 14:53:13.818403 23441 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:53:13.804860 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.60:38493 (local address 127.0.0.1:59314)
0620 14:53:13.805140 (+ 280us) negotiation.cc:107] Waiting for socket to connect
0620 14:53:13.805160 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0620 14:53:13.805270 (+ 110us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 14:53:13.805948 (+ 678us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 14:53:13.805961 (+ 13us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 14:53:13.806461 (+ 500us) client_negotiation.cc:189] Negotiated authn=SASL
0620 14:53:13.806644 (+ 183us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:53:13.806653 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:53:13.808001 (+ 1348us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:53:13.808005 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:53:13.808865 (+ 860us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:53:13.808876 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:53:13.809576 (+ 700us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:53:13.809583 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:53:13.809769 (+ 186us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:53:13.810425 (+ 656us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 14:53:13.810457 (+ 32us) client_negotiation.cc:656] calling sasl_client_start()
0620 14:53:13.812870 (+ 2413us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 14:53:13.815732 (+ 2862us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 14:53:13.815738 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 14:53:13.815742 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0620 14:53:13.816114 (+ 372us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 14:53:13.816534 (+ 420us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 14:53:13.816538 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 14:53:13.816541 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0620 14:53:13.816678 (+ 137us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 14:53:13.817317 (+ 639us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 14:53:13.817321 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 14:53:13.817476 (+ 155us) client_negotiation.cc:769] Sending connection context
0620 14:53:13.817621 (+ 145us) client_negotiation.cc:240] Negotiation successful
0620 14:53:13.817974 (+ 353us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":175,"mutex_wait_us":3}
I20260620 14:53:13.822548 23366 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:53:13.807147 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54628 (local address 127.0.97.61:38589)
0620 14:53:13.807309 (+ 162us) server_negotiation.cc:206] Beginning negotiation
0620 14:53:13.807314 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 14:53:13.807328 (+ 14us) server_negotiation.cc:407] Connection header received
0620 14:53:13.807490 (+ 162us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:53:13.807494 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:53:13.807549 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:53:13.807698 (+ 149us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:53:13.808992 (+ 1294us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:53:13.809866 (+ 874us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:53:13.811297 (+ 1431us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:53:13.811941 (+ 644us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:53:13.812097 (+ 156us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:53:13.816517 (+ 4420us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 14:53:13.816525 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 14:53:13.816530 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 14:53:13.816573 (+ 43us) server_negotiation.cc:922] calling sasl_server_start()
0620 14:53:13.819739 (+ 3166us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:53:13.820509 (+ 770us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:53:13.820515 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:53:13.820519 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:53:13.820659 (+ 140us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:53:13.821073 (+ 414us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:53:13.821079 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:53:13.821082 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:53:13.821352 (+ 270us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 14:53:13.821658 (+ 306us) server_negotiation.cc:1035] Waiting for connection context
0620 14:53:13.821963 (+ 305us) server_negotiation.cc:299] Negotiation successful
0620 14:53:13.822211 (+ 248us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":60}
I20260620 14:53:13.822866 23287 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.97.60" port: 38493 } attempt: 4
I20260620 14:53:13.823953 23436 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 53df900c7e424f3e9824e63bd97f4bf7: Bootstrap starting.
I20260620 14:53:13.828435 23436 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 53df900c7e424f3e9824e63bd97f4bf7: Neither blocks nor log segments found. Creating new log.
I20260620 14:53:13.828533 388 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260620 14:53:13.829928 23436 log.cc:824] T 00000000000000000000000000000000 P 53df900c7e424f3e9824e63bd97f4bf7: Log is configured to *not* fsync() on all Append() calls
Jun 20 14:53:13 dist-test-slave-q9j6 krb5kdc[23207](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781967192, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.62@KRBTEST.COM for kudu/127.0.97.60@KRBTEST.COM
I20260620 14:53:13.833884 23436 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 53df900c7e424f3e9824e63bd97f4bf7: No bootstrap required, opened a new log
I20260620 14:53:13.831823 23262 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "00000000000000000000000000000000"
dest_uuid: "ad601311ce144f35937a457f9694953f"
from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:58820
I20260620 14:53:13.840516 23291 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:53:13.823569 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.60:38493 (local address 127.0.0.1:59330)
0620 14:53:13.823789 (+ 220us) negotiation.cc:107] Waiting for socket to connect
0620 14:53:13.823810 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0620 14:53:13.823928 (+ 118us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 14:53:13.824302 (+ 374us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 14:53:13.824309 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 14:53:13.824832 (+ 523us) client_negotiation.cc:189] Negotiated authn=SASL
0620 14:53:13.825005 (+ 173us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:53:13.825016 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:53:13.826392 (+ 1376us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:53:13.826400 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:53:13.827821 (+ 1421us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:53:13.827836 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:53:13.828763 (+ 927us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:53:13.828769 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:53:13.828941 (+ 172us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:53:13.829619 (+ 678us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 14:53:13.829637 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0620 14:53:13.832564 (+ 2927us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 14:53:13.837396 (+ 4832us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 14:53:13.837404 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 14:53:13.837408 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0620 14:53:13.838013 (+ 605us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 14:53:13.838497 (+ 484us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 14:53:13.838502 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 14:53:13.838505 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0620 14:53:13.838679 (+ 174us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 14:53:13.839409 (+ 730us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 14:53:13.839415 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 14:53:13.839585 (+ 170us) client_negotiation.cc:769] Sending connection context
0620 14:53:13.839745 (+ 160us) client_negotiation.cc:240] Negotiation successful
0620 14:53:13.840069 (+ 324us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":93}
I20260620 14:53:13.840555 23439 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:53:13.823587 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59330 (local address 127.0.97.60:38493)
0620 14:53:13.823808 (+ 221us) server_negotiation.cc:206] Beginning negotiation
0620 14:53:13.823814 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 14:53:13.824000 (+ 186us) server_negotiation.cc:407] Connection header received
0620 14:53:13.824060 (+ 60us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:53:13.824065 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:53:13.824139 (+ 74us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:53:13.824238 (+ 99us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:53:13.825157 (+ 919us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:53:13.826245 (+ 1088us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:53:13.827970 (+ 1725us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:53:13.828476 (+ 506us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:53:13.828626 (+ 150us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:53:13.832829 (+ 4203us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 14:53:13.832837 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 14:53:13.832841 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 14:53:13.832879 (+ 38us) server_negotiation.cc:922] calling sasl_server_start()
0620 14:53:13.837200 (+ 4321us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:53:13.838178 (+ 978us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:53:13.838184 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:53:13.838187 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:53:13.838364 (+ 177us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:53:13.838807 (+ 443us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:53:13.838812 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:53:13.838816 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:53:13.839267 (+ 451us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 14:53:13.839445 (+ 178us) server_negotiation.cc:1035] Waiting for connection context
0620 14:53:13.839892 (+ 447us) server_negotiation.cc:299] Negotiation successful
0620 14:53:13.840144 (+ 252us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":38}
I20260620 14:53:13.845180 23436 raft_consensus.cc:348] T 00000000000000000000000000000000 P 53df900c7e424f3e9824e63bd97f4bf7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ad601311ce144f35937a457f9694953f" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 35275 } } peers { permanent_uuid: "00c5d93a338047ba8d85935a5cbeab53" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 38589 } } peers { permanent_uuid: "53df900c7e424f3e9824e63bd97f4bf7" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 38493 } }
I20260620 14:53:13.845677 23436 raft_consensus.cc:729] T 00000000000000000000000000000000 P 53df900c7e424f3e9824e63bd97f4bf7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 53df900c7e424f3e9824e63bd97f4bf7, State: Initialized, Role: FOLLOWER
I20260620 14:53:13.846256 23436 consensus_queue.cc:260] T 00000000000000000000000000000000 P 53df900c7e424f3e9824e63bd97f4bf7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ad601311ce144f35937a457f9694953f" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 35275 } } peers { permanent_uuid: "00c5d93a338047ba8d85935a5cbeab53" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 38589 } } peers { permanent_uuid: "53df900c7e424f3e9824e63bd97f4bf7" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 38493 } }
I20260620 14:53:13.846757 23287 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ad601311ce144f35937a457f9694953f: Bootstrap starting.
I20260620 14:53:13.847306 23443 sys_catalog.cc:455] T 00000000000000000000000000000000 P 53df900c7e424f3e9824e63bd97f4bf7 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ad601311ce144f35937a457f9694953f" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 35275 } } peers { permanent_uuid: "00c5d93a338047ba8d85935a5cbeab53" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 38589 } } peers { permanent_uuid: "53df900c7e424f3e9824e63bd97f4bf7" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 38493 } } }
I20260620 14:53:13.847726 23443 sys_catalog.cc:458] T 00000000000000000000000000000000 P 53df900c7e424f3e9824e63bd97f4bf7 [sys.catalog]: This master's current role is: FOLLOWER
I20260620 14:53:13.848731 23436 sys_catalog.cc:564] T 00000000000000000000000000000000 P 53df900c7e424f3e9824e63bd97f4bf7 [sys.catalog]: configured and running, proceeding with master startup.
I20260620 14:53:13.851363 23287 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P ad601311ce144f35937a457f9694953f: Neither blocks nor log segments found. Creating new log.
I20260620 14:53:13.852977 23287 log.cc:824] T 00000000000000000000000000000000 P ad601311ce144f35937a457f9694953f: Log is configured to *not* fsync() on all Append() calls
I20260620 14:53:13.856669 23287 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ad601311ce144f35937a457f9694953f: No bootstrap required, opened a new log
W20260620 14:53:13.860661 23455 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 53df900c7e424f3e9824e63bd97f4bf7: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260620 14:53:13.860908 23455 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260620 14:53:13.865164 23287 raft_consensus.cc:348] T 00000000000000000000000000000000 P ad601311ce144f35937a457f9694953f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ad601311ce144f35937a457f9694953f" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 35275 } } peers { permanent_uuid: "00c5d93a338047ba8d85935a5cbeab53" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 38589 } } peers { permanent_uuid: "53df900c7e424f3e9824e63bd97f4bf7" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 38493 } }
I20260620 14:53:13.865667 23287 raft_consensus.cc:729] T 00000000000000000000000000000000 P ad601311ce144f35937a457f9694953f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ad601311ce144f35937a457f9694953f, State: Initialized, Role: FOLLOWER
I20260620 14:53:13.866271 23287 consensus_queue.cc:260] T 00000000000000000000000000000000 P ad601311ce144f35937a457f9694953f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ad601311ce144f35937a457f9694953f" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 35275 } } peers { permanent_uuid: "00c5d93a338047ba8d85935a5cbeab53" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 38589 } } peers { permanent_uuid: "53df900c7e424f3e9824e63bd97f4bf7" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 38493 } }
I20260620 14:53:13.867722 23457 sys_catalog.cc:455] T 00000000000000000000000000000000 P ad601311ce144f35937a457f9694953f [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ad601311ce144f35937a457f9694953f" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 35275 } } peers { permanent_uuid: "00c5d93a338047ba8d85935a5cbeab53" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 38589 } } peers { permanent_uuid: "53df900c7e424f3e9824e63bd97f4bf7" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 38493 } } }
I20260620 14:53:13.868485 23457 sys_catalog.cc:458] T 00000000000000000000000000000000 P ad601311ce144f35937a457f9694953f [sys.catalog]: This master's current role is: FOLLOWER
I20260620 14:53:13.856859 23369 master_runner.cc:186] Error getting master registration for 127.0.97.62:35275: OK, instance_id { permanent_uuid: "ad601311ce144f35937a457f9694953f" instance_seqno: 1781967192670006 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260620 14:53:13.868928 23369 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260620 14:53:13.868988 23287 sys_catalog.cc:564] T 00000000000000000000000000000000 P ad601311ce144f35937a457f9694953f [sys.catalog]: configured and running, proceeding with master startup.
I20260620 14:53:13.869095 23262 raft_consensus.cc:482] T 00000000000000000000000000000000 P ad601311ce144f35937a457f9694953f [term 0 FOLLOWER]: Starting forced leader election (received explicit request)
I20260620 14:53:13.869297 23262 raft_consensus.cc:3037] T 00000000000000000000000000000000 P ad601311ce144f35937a457f9694953f [term 0 FOLLOWER]: Advancing to term 1
I20260620 14:53:13.873409 23262 raft_consensus.cc:504] T 00000000000000000000000000000000 P ad601311ce144f35937a457f9694953f [term 1 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ad601311ce144f35937a457f9694953f" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 35275 } } peers { permanent_uuid: "00c5d93a338047ba8d85935a5cbeab53" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 38589 } } peers { permanent_uuid: "53df900c7e424f3e9824e63bd97f4bf7" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 38493 } }
I20260620 14:53:13.876209 23411 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "ad601311ce144f35937a457f9694953f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: true dest_uuid: "53df900c7e424f3e9824e63bd97f4bf7"
I20260620 14:53:13.876752 23411 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 53df900c7e424f3e9824e63bd97f4bf7 [term 0 FOLLOWER]: Advancing to term 1
I20260620 14:53:13.880661 23411 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 53df900c7e424f3e9824e63bd97f4bf7 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ad601311ce144f35937a457f9694953f in term 1.
I20260620 14:53:13.881560 23236 leader_election.cc:302] T 00000000000000000000000000000000 P ad601311ce144f35937a457f9694953f [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 53df900c7e424f3e9824e63bd97f4bf7, ad601311ce144f35937a457f9694953f; no voters:
I20260620 14:53:13.882259 23262 leader_election.cc:288] T 00000000000000000000000000000000 P ad601311ce144f35937a457f9694953f [CANDIDATE]: Term 1 election: Requested vote from peers 00c5d93a338047ba8d85935a5cbeab53 (127.0.97.61:38589), 53df900c7e424f3e9824e63bd97f4bf7 (127.0.97.60:38493)
I20260620 14:53:13.882900 23457 raft_consensus.cc:2781] T 00000000000000000000000000000000 P ad601311ce144f35937a457f9694953f [term 1 FOLLOWER]: Leader election won for term 1
I20260620 14:53:13.903223 23457 raft_consensus.cc:686] T 00000000000000000000000000000000 P ad601311ce144f35937a457f9694953f [term 1 LEADER]: Becoming Leader. State: Replica: ad601311ce144f35937a457f9694953f, State: Running, Role: LEADER
I20260620 14:53:13.904119 23457 consensus_queue.cc:237] T 00000000000000000000000000000000 P ad601311ce144f35937a457f9694953f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ad601311ce144f35937a457f9694953f" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 35275 } } peers { permanent_uuid: "00c5d93a338047ba8d85935a5cbeab53" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 38589 } } peers { permanent_uuid: "53df900c7e424f3e9824e63bd97f4bf7" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 38493 } }
W20260620 14:53:13.908046 23241 tablet.cc:2369] T 00000000000000000000000000000000 P ad601311ce144f35937a457f9694953f: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260620 14:53:13.925570 23465 sys_catalog.cc:455] T 00000000000000000000000000000000 P ad601311ce144f35937a457f9694953f [sys.catalog]: SysCatalogTable state changed. Reason: New leader ad601311ce144f35937a457f9694953f. Latest consensus state: current_term: 1 leader_uuid: "ad601311ce144f35937a457f9694953f" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ad601311ce144f35937a457f9694953f" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 35275 } } peers { permanent_uuid: "00c5d93a338047ba8d85935a5cbeab53" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 38589 } } peers { permanent_uuid: "53df900c7e424f3e9824e63bd97f4bf7" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 38493 } } }
I20260620 14:53:13.926056 23465 sys_catalog.cc:458] T 00000000000000000000000000000000 P ad601311ce144f35937a457f9694953f [sys.catalog]: This master's current role is: LEADER
I20260620 14:53:13.939664 23361 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.97.60" port: 38493 } attempt: 5
I20260620 14:53:13.947839 23439 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:53:13.912430 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59344 (local address 127.0.97.60:38493)
0620 14:53:13.912871 (+ 441us) server_negotiation.cc:206] Beginning negotiation
0620 14:53:13.912877 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 14:53:13.920970 (+ 8093us) server_negotiation.cc:407] Connection header received
0620 14:53:13.921219 (+ 249us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:53:13.921226 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:53:13.921359 (+ 133us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:53:13.921486 (+ 127us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:53:13.925610 (+ 4124us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:53:13.926703 (+ 1093us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:53:13.930075 (+ 3372us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:53:13.930817 (+ 742us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:53:13.931325 (+ 508us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:53:13.938270 (+ 6945us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 14:53:13.938278 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 14:53:13.938283 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 14:53:13.938334 (+ 51us) server_negotiation.cc:922] calling sasl_server_start()
0620 14:53:13.942391 (+ 4057us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:53:13.944009 (+ 1618us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:53:13.944018 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:53:13.944028 (+ 10us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:53:13.944215 (+ 187us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:53:13.944942 (+ 727us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:53:13.944949 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:53:13.944954 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:53:13.945329 (+ 375us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 14:53:13.945595 (+ 266us) server_negotiation.cc:1035] Waiting for connection context
0620 14:53:13.946589 (+ 994us) server_negotiation.cc:299] Negotiation successful
0620 14:53:13.947058 (+ 469us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":318}
I20260620 14:53:13.950875 23478 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 14:53:13.951427 23440 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:53:13.912068 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58834 (local address 127.0.97.62:35275)
0620 14:53:13.912711 (+ 643us) server_negotiation.cc:206] Beginning negotiation
0620 14:53:13.912721 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0620 14:53:13.921936 (+ 9215us) server_negotiation.cc:407] Connection header received
0620 14:53:13.922217 (+ 281us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:53:13.922224 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:53:13.922308 (+ 84us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:53:13.922377 (+ 69us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:53:13.924799 (+ 2422us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:53:13.926058 (+ 1259us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:53:13.928715 (+ 2657us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:53:13.929658 (+ 943us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:53:13.929796 (+ 138us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:53:13.934719 (+ 4923us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 14:53:13.934730 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 14:53:13.934738 (+ 8us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 14:53:13.934818 (+ 80us) server_negotiation.cc:922] calling sasl_server_start()
0620 14:53:13.939556 (+ 4738us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:53:13.947504 (+ 7948us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:53:13.947515 (+ 11us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:53:13.947521 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:53:13.947792 (+ 271us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:53:13.948674 (+ 882us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:53:13.948683 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:53:13.948689 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:53:13.949180 (+ 491us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 14:53:13.949523 (+ 343us) server_negotiation.cc:1035] Waiting for connection context
0620 14:53:13.950682 (+ 1159us) server_negotiation.cc:299] Negotiation successful
0620 14:53:13.950961 (+ 279us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":460}
I20260620 14:53:13.956494 23477 catalog_manager.cc:795] Waiting for catalog manager background task thread to start: Service unavailable: Catalog manager is not initialized. State: Starting
I20260620 14:53:13.961985 23478 catalog_manager.cc:1484] Initializing Kudu cluster ID...
Jun 20 14:53:13 dist-test-slave-q9j6 krb5kdc[23207](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781967193, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.61@KRBTEST.COM for kudu/127.0.97.60@KRBTEST.COM
I20260620 14:53:13.979805 23366 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:53:13.922473 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54642 (local address 127.0.97.61:38589)
0620 14:53:13.922662 (+ 189us) server_negotiation.cc:206] Beginning negotiation
0620 14:53:13.922668 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 14:53:13.952357 (+ 29689us) server_negotiation.cc:407] Connection header received
0620 14:53:13.953602 (+ 1245us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:53:13.953609 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:53:13.953708 (+ 99us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:53:13.953782 (+ 74us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:53:13.961079 (+ 7297us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:53:13.962225 (+ 1146us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:53:13.967866 (+ 5641us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:53:13.968641 (+ 775us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:53:13.968828 (+ 187us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:53:13.972798 (+ 3970us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 14:53:13.972803 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 14:53:13.972806 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 14:53:13.972842 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0620 14:53:13.976488 (+ 3646us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:53:13.977505 (+ 1017us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:53:13.977511 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:53:13.977515 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:53:13.977695 (+ 180us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:53:13.978164 (+ 469us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:53:13.978170 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:53:13.978173 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:53:13.978498 (+ 325us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 14:53:13.978791 (+ 293us) server_negotiation.cc:1035] Waiting for connection context
0620 14:53:13.979136 (+ 345us) server_negotiation.cc:299] Negotiation successful
0620 14:53:13.979367 (+ 231us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":75}
W20260620 14:53:13.983152 23390 tablet.cc:2369] T 00000000000000000000000000000000 P 53df900c7e424f3e9824e63bd97f4bf7: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260620 14:53:13.961359 23220 master_runner.cc:186] Error getting master registration for 127.0.97.61:38589: OK, instance_id { permanent_uuid: "00c5d93a338047ba8d85935a5cbeab53" instance_seqno: 1781967193207350 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260620 14:53:13.988674 23220 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260620 14:53:13.998425 23364 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:53:13.940382 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.60:38493 (local address 127.0.0.1:59356)
0620 14:53:13.940714 (+ 332us) negotiation.cc:107] Waiting for socket to connect
0620 14:53:13.940734 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0620 14:53:13.940871 (+ 137us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 14:53:13.949136 (+ 8265us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 14:53:13.949145 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 14:53:13.949719 (+ 574us) client_negotiation.cc:189] Negotiated authn=SASL
0620 14:53:13.949993 (+ 274us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:53:13.950008 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:53:13.952731 (+ 2723us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:53:13.952737 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:53:13.954208 (+ 1471us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:53:13.954226 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:53:13.955903 (+ 1677us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:53:13.955910 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:53:13.956149 (+ 239us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:53:13.957051 (+ 902us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 14:53:13.957078 (+ 27us) client_negotiation.cc:656] calling sasl_client_start()
0620 14:53:13.971962 (+ 14884us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 14:53:13.986773 (+ 14811us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 14:53:13.986780 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 14:53:13.986785 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0620 14:53:13.987415 (+ 630us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 14:53:13.996586 (+ 9171us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 14:53:13.996593 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 14:53:13.996597 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0620 14:53:13.996818 (+ 221us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 14:53:13.997564 (+ 746us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 14:53:13.997571 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 14:53:13.997744 (+ 173us) client_negotiation.cc:769] Sending connection context
0620 14:53:13.997866 (+ 122us) client_negotiation.cc:240] Negotiation successful
0620 14:53:13.998148 (+ 282us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":194}
I20260620 14:53:13.999552 23411 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 53df900c7e424f3e9824e63bd97f4bf7 [term 1 FOLLOWER]: Refusing update from remote peer ad601311ce144f35937a457f9694953f: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 14:53:13.999773 23439 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:53:13.940484 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59356 (local address 127.0.97.60:38493)
0620 14:53:13.948318 (+ 7834us) server_negotiation.cc:206] Beginning negotiation
0620 14:53:13.948324 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 14:53:13.948375 (+ 51us) server_negotiation.cc:407] Connection header received
0620 14:53:13.948463 (+ 88us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:53:13.948498 (+ 35us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:53:13.948823 (+ 325us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:53:13.950264 (+ 1441us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:53:13.950549 (+ 285us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:53:13.952430 (+ 1881us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:53:13.954499 (+ 2069us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:53:13.955645 (+ 1146us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:53:13.979805 (+ 24160us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:53:13.980393 (+ 588us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 14:53:13.980400 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 14:53:13.980408 (+ 8us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 14:53:13.980450 (+ 42us) server_negotiation.cc:922] calling sasl_server_start()
0620 14:53:13.986562 (+ 6112us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:53:13.995707 (+ 9145us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:53:13.995714 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:53:13.995719 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:53:13.996404 (+ 685us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:53:13.996992 (+ 588us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:53:13.996997 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:53:13.997001 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:53:13.997416 (+ 415us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 14:53:13.997579 (+ 163us) server_negotiation.cc:1035] Waiting for connection context
0620 14:53:13.998934 (+ 1355us) server_negotiation.cc:299] Negotiation successful
0620 14:53:13.999251 (+ 317us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":7613,"thread_start_us":27817,"threads_started":1}
I20260620 14:53:14.001075 23465 consensus_queue.cc:1035] T 00000000000000000000000000000000 P ad601311ce144f35937a457f9694953f [LEADER]: Connected to new peer: Peer: permanent_uuid: "53df900c7e424f3e9824e63bd97f4bf7" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 38493 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 14:53:14.006269 23361 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 00c5d93a338047ba8d85935a5cbeab53: Bootstrap starting.
I20260620 14:53:14.017163 23361 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 00c5d93a338047ba8d85935a5cbeab53: Neither blocks nor log segments found. Creating new log.
I20260620 14:53:14.021122 23361 log.cc:824] T 00000000000000000000000000000000 P 00c5d93a338047ba8d85935a5cbeab53: Log is configured to *not* fsync() on all Append() calls
I20260620 14:53:14.027171 23361 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 00c5d93a338047ba8d85935a5cbeab53: No bootstrap required, opened a new log
I20260620 14:53:14.029603 23443 sys_catalog.cc:455] T 00000000000000000000000000000000 P 53df900c7e424f3e9824e63bd97f4bf7 [sys.catalog]: SysCatalogTable state changed. Reason: New leader ad601311ce144f35937a457f9694953f. Latest consensus state: current_term: 1 leader_uuid: "ad601311ce144f35937a457f9694953f" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ad601311ce144f35937a457f9694953f" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 35275 } } peers { permanent_uuid: "00c5d93a338047ba8d85935a5cbeab53" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 38589 } } peers { permanent_uuid: "53df900c7e424f3e9824e63bd97f4bf7" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 38493 } } }
I20260620 14:53:14.029976 23443 sys_catalog.cc:458] T 00000000000000000000000000000000 P 53df900c7e424f3e9824e63bd97f4bf7 [sys.catalog]: This master's current role is: FOLLOWER
I20260620 14:53:14.048888 23457 sys_catalog.cc:455] T 00000000000000000000000000000000 P ad601311ce144f35937a457f9694953f [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "ad601311ce144f35937a457f9694953f" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ad601311ce144f35937a457f9694953f" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 35275 } } peers { permanent_uuid: "00c5d93a338047ba8d85935a5cbeab53" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 38589 } } peers { permanent_uuid: "53df900c7e424f3e9824e63bd97f4bf7" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 38493 } } }
I20260620 14:53:14.049278 23457 sys_catalog.cc:458] T 00000000000000000000000000000000 P ad601311ce144f35937a457f9694953f [sys.catalog]: This master's current role is: LEADER
I20260620 14:53:14.050143 23478 catalog_manager.cc:1347] Generated new cluster ID: ea5e8f73ce4e457492219b4198fcc62c
I20260620 14:53:14.050287 23478 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 14:53:14.054415 23443 sys_catalog.cc:455] T 00000000000000000000000000000000 P 53df900c7e424f3e9824e63bd97f4bf7 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "ad601311ce144f35937a457f9694953f" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ad601311ce144f35937a457f9694953f" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 35275 } } peers { permanent_uuid: "00c5d93a338047ba8d85935a5cbeab53" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 38589 } } peers { permanent_uuid: "53df900c7e424f3e9824e63bd97f4bf7" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 38493 } } }
I20260620 14:53:14.054744 23443 sys_catalog.cc:458] T 00000000000000000000000000000000 P 53df900c7e424f3e9824e63bd97f4bf7 [sys.catalog]: This master's current role is: FOLLOWER
I20260620 14:53:14.060271 23361 raft_consensus.cc:348] T 00000000000000000000000000000000 P 00c5d93a338047ba8d85935a5cbeab53 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ad601311ce144f35937a457f9694953f" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 35275 } } peers { permanent_uuid: "00c5d93a338047ba8d85935a5cbeab53" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 38589 } } peers { permanent_uuid: "53df900c7e424f3e9824e63bd97f4bf7" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 38493 } }
I20260620 14:53:14.060896 23361 raft_consensus.cc:729] T 00000000000000000000000000000000 P 00c5d93a338047ba8d85935a5cbeab53 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 00c5d93a338047ba8d85935a5cbeab53, State: Initialized, Role: FOLLOWER
I20260620 14:53:14.065691 23361 consensus_queue.cc:260] T 00000000000000000000000000000000 P 00c5d93a338047ba8d85935a5cbeab53 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ad601311ce144f35937a457f9694953f" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 35275 } } peers { permanent_uuid: "00c5d93a338047ba8d85935a5cbeab53" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 38589 } } peers { permanent_uuid: "53df900c7e424f3e9824e63bd97f4bf7" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 38493 } }
I20260620 14:53:14.073988 23487 sys_catalog.cc:455] T 00000000000000000000000000000000 P 00c5d93a338047ba8d85935a5cbeab53 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ad601311ce144f35937a457f9694953f" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 35275 } } peers { permanent_uuid: "00c5d93a338047ba8d85935a5cbeab53" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 38589 } } peers { permanent_uuid: "53df900c7e424f3e9824e63bd97f4bf7" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 38493 } } }
I20260620 14:53:14.074901 23487 sys_catalog.cc:458] T 00000000000000000000000000000000 P 00c5d93a338047ba8d85935a5cbeab53 [sys.catalog]: This master's current role is: FOLLOWER
I20260620 14:53:14.079048 23361 sys_catalog.cc:564] T 00000000000000000000000000000000 P 00c5d93a338047ba8d85935a5cbeab53 [sys.catalog]: configured and running, proceeding with master startup.
I20260620 14:53:14.079896 23335 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 00c5d93a338047ba8d85935a5cbeab53 [term 0 FOLLOWER]: Advancing to term 1
I20260620 14:53:14.083456 23336 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "ad601311ce144f35937a457f9694953f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: true dest_uuid: "00c5d93a338047ba8d85935a5cbeab53"
I20260620 14:53:14.085273 23478 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 14:53:14.087452 23478 catalog_manager.cc:1504] Loading token signing keys...
I20260620 14:53:14.092108 23335 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 00c5d93a338047ba8d85935a5cbeab53 [term 1 FOLLOWER]: Refusing update from remote peer ad601311ce144f35937a457f9694953f: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 14:53:14.110741 23336 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 00c5d93a338047ba8d85935a5cbeab53 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ad601311ce144f35937a457f9694953f in term 1.
I20260620 14:53:14.112102 23465 consensus_queue.cc:1035] T 00000000000000000000000000000000 P ad601311ce144f35937a457f9694953f [LEADER]: Connected to new peer: Peer: permanent_uuid: "00c5d93a338047ba8d85935a5cbeab53" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 38589 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 14:53:14.130200 23478 catalog_manager.cc:5868] T 00000000000000000000000000000000 P ad601311ce144f35937a457f9694953f: Generated new TSK 0
I20260620 14:53:14.131376 23478 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260620 14:53:14.188169 23487 sys_catalog.cc:455] T 00000000000000000000000000000000 P 00c5d93a338047ba8d85935a5cbeab53 [sys.catalog]: SysCatalogTable state changed. Reason: New leader ad601311ce144f35937a457f9694953f. Latest consensus state: current_term: 1 leader_uuid: "ad601311ce144f35937a457f9694953f" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ad601311ce144f35937a457f9694953f" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 35275 } } peers { permanent_uuid: "00c5d93a338047ba8d85935a5cbeab53" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 38589 } } peers { permanent_uuid: "53df900c7e424f3e9824e63bd97f4bf7" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 38493 } } }
I20260620 14:53:14.188524 23487 sys_catalog.cc:458] T 00000000000000000000000000000000 P 00c5d93a338047ba8d85935a5cbeab53 [sys.catalog]: This master's current role is: FOLLOWER
I20260620 14:53:14.188644 23457 sys_catalog.cc:455] T 00000000000000000000000000000000 P ad601311ce144f35937a457f9694953f [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "ad601311ce144f35937a457f9694953f" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ad601311ce144f35937a457f9694953f" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 35275 } } peers { permanent_uuid: "00c5d93a338047ba8d85935a5cbeab53" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 38589 } } peers { permanent_uuid: "53df900c7e424f3e9824e63bd97f4bf7" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 38493 } } }
I20260620 14:53:14.188951 23457 sys_catalog.cc:458] T 00000000000000000000000000000000 P ad601311ce144f35937a457f9694953f [sys.catalog]: This master's current role is: LEADER
I20260620 14:53:14.190064 23500 sys_catalog.cc:455] T 00000000000000000000000000000000 P 00c5d93a338047ba8d85935a5cbeab53 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "ad601311ce144f35937a457f9694953f" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ad601311ce144f35937a457f9694953f" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 35275 } } peers { permanent_uuid: "00c5d93a338047ba8d85935a5cbeab53" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 38589 } } peers { permanent_uuid: "53df900c7e424f3e9824e63bd97f4bf7" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 38493 } } }
I20260620 14:53:14.190431 23500 sys_catalog.cc:458] T 00000000000000000000000000000000 P 00c5d93a338047ba8d85935a5cbeab53 [sys.catalog]: This master's current role is: FOLLOWER
I20260620 14:53:14.200780 23504 catalog_manager.cc:795] Waiting for catalog manager background task thread to start: Service unavailable: Catalog manager is not initialized. State: Starting
W20260620 14:53:14.251480 23440 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 14:53:14.246917 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58850 (local address 127.0.97.62:35275)
0620 14:53:14.247121 (+ 204us) server_negotiation.cc:206] Beginning negotiation
0620 14:53:14.247128 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0620 14:53:14.249506 (+ 2378us) server_negotiation.cc:407] Connection header received
0620 14:53:14.249809 (+ 303us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:53:14.249817 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:53:14.249911 (+ 94us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:53:14.250002 (+ 91us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:53:14.251198 (+ 1196us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:58850: BlockingRecv error: recv got EOF from 127.0.0.1:58850 (error 108)
Metrics: {"server-negotiator.queue_time_us":72}
W20260620 14:53:14.253044 23439 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 14:53:14.247357 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59364 (local address 127.0.97.60:38493)
0620 14:53:14.247571 (+ 214us) server_negotiation.cc:206] Beginning negotiation
0620 14:53:14.247580 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0620 14:53:14.249190 (+ 1610us) server_negotiation.cc:407] Connection header received
0620 14:53:14.250538 (+ 1348us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:53:14.250544 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:53:14.250657 (+ 113us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:53:14.250728 (+ 71us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:53:14.252762 (+ 2034us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:59364: BlockingRecv error: recv got EOF from 127.0.0.1:59364 (error 108)
Metrics: {"server-negotiator.queue_time_us":49}
W20260620 14:53:14.254550 23366 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 14:53:14.249735 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54658 (local address 127.0.97.61:38589)
0620 14:53:14.249932 (+ 197us) server_negotiation.cc:206] Beginning negotiation
0620 14:53:14.249938 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 14:53:14.252387 (+ 2449us) server_negotiation.cc:407] Connection header received
0620 14:53:14.252652 (+ 265us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:53:14.252658 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:53:14.252764 (+ 106us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:53:14.252845 (+ 81us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:53:14.254319 (+ 1474us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:54658: BlockingRecv error: recv got EOF from 127.0.0.1:54658 (error 108)
Metrics: {"server-negotiator.queue_time_us":62}
W20260620 14:53:14.262617 23366 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 14:53:14.260055 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54660 (local address 127.0.97.61:38589)
0620 14:53:14.260286 (+ 231us) server_negotiation.cc:206] Beginning negotiation
0620 14:53:14.260295 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0620 14:53:14.260927 (+ 632us) server_negotiation.cc:407] Connection header received
0620 14:53:14.261193 (+ 266us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:53:14.261199 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:53:14.261344 (+ 145us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:53:14.261463 (+ 119us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:53:14.262323 (+ 860us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:54660: BlockingRecv error: recv got EOF from 127.0.0.1:54660 (error 108)
Metrics: {"server-negotiator.queue_time_us":80}
W20260620 14:53:14.269484 23366 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 14:53:14.267052 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54670 (local address 127.0.97.61:38589)
0620 14:53:14.267228 (+ 176us) server_negotiation.cc:206] Beginning negotiation
0620 14:53:14.267235 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0620 14:53:14.267748 (+ 513us) server_negotiation.cc:407] Connection header received
0620 14:53:14.268063 (+ 315us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:53:14.268069 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:53:14.268168 (+ 99us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:53:14.268244 (+ 76us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:53:14.269255 (+ 1011us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:54670: BlockingRecv error: recv got EOF from 127.0.0.1:54670 (error 108)
Metrics: {"server-negotiator.queue_time_us":60}
W20260620 14:53:14.278644 23366 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 14:53:14.276163 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54684 (local address 127.0.97.61:38589)
0620 14:53:14.276363 (+ 200us) server_negotiation.cc:206] Beginning negotiation
0620 14:53:14.276370 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0620 14:53:14.276892 (+ 522us) server_negotiation.cc:407] Connection header received
0620 14:53:14.277196 (+ 304us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:53:14.277202 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:53:14.277312 (+ 110us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:53:14.277389 (+ 77us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:53:14.278390 (+ 1001us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:54684: BlockingRecv error: recv got EOF from 127.0.0.1:54684 (error 108)
Metrics: {"server-negotiator.queue_time_us":67}
W20260620 14:53:14.294981 23366 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 14:53:14.292616 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54692 (local address 127.0.97.61:38589)
0620 14:53:14.292813 (+ 197us) server_negotiation.cc:206] Beginning negotiation
0620 14:53:14.292821 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0620 14:53:14.293392 (+ 571us) server_negotiation.cc:407] Connection header received
0620 14:53:14.293594 (+ 202us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:53:14.293602 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:53:14.293714 (+ 112us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:53:14.293841 (+ 127us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:53:14.294727 (+ 886us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:54692: BlockingRecv error: recv got EOF from 127.0.0.1:54692 (error 108)
Metrics: {"server-negotiator.queue_time_us":72}
W20260620 14:53:14.315274 23366 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 14:53:14.312861 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54696 (local address 127.0.97.61:38589)
0620 14:53:14.313071 (+ 210us) server_negotiation.cc:206] Beginning negotiation
0620 14:53:14.313078 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0620 14:53:14.313707 (+ 629us) server_negotiation.cc:407] Connection header received
0620 14:53:14.313931 (+ 224us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:53:14.313937 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:53:14.314030 (+ 93us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:53:14.314140 (+ 110us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:53:14.315041 (+ 901us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:54696: BlockingRecv error: recv got EOF from 127.0.0.1:54696 (error 108)
Metrics: {"server-negotiator.queue_time_us":71}
W20260620 14:53:14.337929 23366 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 14:53:14.335343 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54706 (local address 127.0.97.61:38589)
0620 14:53:14.335563 (+ 220us) server_negotiation.cc:206] Beginning negotiation
0620 14:53:14.335571 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0620 14:53:14.336202 (+ 631us) server_negotiation.cc:407] Connection header received
0620 14:53:14.336441 (+ 239us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:53:14.336447 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:53:14.336547 (+ 100us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:53:14.336676 (+ 129us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:53:14.337650 (+ 974us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:54706: BlockingRecv error: recv got EOF from 127.0.0.1:54706 (error 108)
Metrics: {"server-negotiator.queue_time_us":75}
W20260620 14:53:14.380005 23366 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 14:53:14.377469 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54714 (local address 127.0.97.61:38589)
0620 14:53:14.377674 (+ 205us) server_negotiation.cc:206] Beginning negotiation
0620 14:53:14.377683 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0620 14:53:14.378252 (+ 569us) server_negotiation.cc:407] Connection header received
0620 14:53:14.378498 (+ 246us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:53:14.378505 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:53:14.378615 (+ 110us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:53:14.378741 (+ 126us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:53:14.379714 (+ 973us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:54714: BlockingRecv error: recv got EOF from 127.0.0.1:54714 (error 108)
Metrics: {"server-negotiator.queue_time_us":77}
W20260620 14:53:14.449797 23366 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 14:53:14.447666 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54724 (local address 127.0.97.61:38589)
0620 14:53:14.447906 (+ 240us) server_negotiation.cc:206] Beginning negotiation
0620 14:53:14.447919 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0620 14:53:14.448248 (+ 329us) server_negotiation.cc:407] Connection header received
0620 14:53:14.448479 (+ 231us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:53:14.448484 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:53:14.448557 (+ 73us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:53:14.448649 (+ 92us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:53:14.449548 (+ 899us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:54724: BlockingRecv error: recv got EOF from 127.0.0.1:54724 (error 108)
Metrics: {"server-negotiator.queue_time_us":87}
W20260620 14:53:14.591362 23366 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 14:53:14.589233 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54740 (local address 127.0.97.61:38589)
0620 14:53:14.589507 (+ 274us) server_negotiation.cc:206] Beginning negotiation
0620 14:53:14.589518 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0620 14:53:14.589885 (+ 367us) server_negotiation.cc:407] Connection header received
0620 14:53:14.590142 (+ 257us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:53:14.590146 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:53:14.590220 (+ 74us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:53:14.590341 (+ 121us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:53:14.591136 (+ 795us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:54740: BlockingRecv error: recv got EOF from 127.0.0.1:54740 (error 108)
Metrics: {"server-negotiator.queue_time_us":76}
W20260620 14:53:14.854097 23366 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 14:53:14.852137 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54754 (local address 127.0.97.61:38589)
0620 14:53:14.852425 (+ 288us) server_negotiation.cc:206] Beginning negotiation
0620 14:53:14.852432 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0620 14:53:14.852652 (+ 220us) server_negotiation.cc:407] Connection header received
0620 14:53:14.852907 (+ 255us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:53:14.852912 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:53:14.852984 (+ 72us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:53:14.853097 (+ 113us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:53:14.853871 (+ 774us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:54754: BlockingRecv error: recv got EOF from 127.0.0.1:54754 (error 108)
Metrics: {"server-negotiator.queue_time_us":77}
I20260620 14:53:14.862920 23455 catalog_manager.cc:1259] Loaded cluster ID: ea5e8f73ce4e457492219b4198fcc62c
I20260620 14:53:14.863032 23455 catalog_manager.cc:1552] T 00000000000000000000000000000000 P 53df900c7e424f3e9824e63bd97f4bf7: loading cluster ID for follower catalog manager: success
I20260620 14:53:14.865577 23455 catalog_manager.cc:1574] T 00000000000000000000000000000000 P 53df900c7e424f3e9824e63bd97f4bf7: acquiring CA information for follower catalog manager: success
I20260620 14:53:14.867420 23455 catalog_manager.cc:1602] T 00000000000000000000000000000000 P 53df900c7e424f3e9824e63bd97f4bf7: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260620 14:53:15.205616 23504 catalog_manager.cc:1259] Loaded cluster ID: ea5e8f73ce4e457492219b4198fcc62c
I20260620 14:53:15.205752 23504 catalog_manager.cc:1552] T 00000000000000000000000000000000 P 00c5d93a338047ba8d85935a5cbeab53: loading cluster ID for follower catalog manager: success
I20260620 14:53:15.209089 23504 catalog_manager.cc:1574] T 00000000000000000000000000000000 P 00c5d93a338047ba8d85935a5cbeab53: acquiring CA information for follower catalog manager: success
I20260620 14:53:15.211463 23504 catalog_manager.cc:1602] T 00000000000000000000000000000000 P 00c5d93a338047ba8d85935a5cbeab53: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260620 14:53:15.378858 23569 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:53:15.369851 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54762 (local address 127.0.97.61:38589)
0620 14:53:15.372707 (+ 2856us) server_negotiation.cc:206] Beginning negotiation
0620 14:53:15.372712 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 14:53:15.372731 (+ 19us) server_negotiation.cc:407] Connection header received
0620 14:53:15.372802 (+ 71us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:53:15.372806 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:53:15.372965 (+ 159us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:53:15.373092 (+ 127us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 14:53:15.373644 (+ 552us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:53:15.374519 (+ 875us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:53:15.376428 (+ 1909us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:53:15.376866 (+ 438us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:53:15.376962 (+ 96us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:53:15.377581 (+ 619us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 14:53:15.378170 (+ 589us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0620 14:53:15.378287 (+ 117us) server_negotiation.cc:1035] Waiting for connection context
0620 14:53:15.378429 (+ 142us) server_negotiation.cc:299] Negotiation successful
0620 14:53:15.378519 (+ 90us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":2754,"thread_start_us":2593,"threads_started":1}
I20260620 14:53:15.392920 23575 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:53:15.385753 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59374 (local address 127.0.97.60:38493)
0620 14:53:15.386307 (+ 554us) server_negotiation.cc:206] Beginning negotiation
0620 14:53:15.386314 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0620 14:53:15.386650 (+ 336us) server_negotiation.cc:407] Connection header received
0620 14:53:15.386905 (+ 255us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:53:15.386909 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:53:15.387068 (+ 159us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:53:15.387217 (+ 149us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 14:53:15.387768 (+ 551us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:53:15.388483 (+ 715us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:53:15.390470 (+ 1987us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:53:15.390861 (+ 391us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:53:15.390955 (+ 94us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:53:15.391581 (+ 626us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 14:53:15.392223 (+ 642us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0620 14:53:15.392343 (+ 120us) server_negotiation.cc:1035] Waiting for connection context
0620 14:53:15.392491 (+ 148us) server_negotiation.cc:299] Negotiation successful
0620 14:53:15.392566 (+ 75us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":394,"thread_start_us":260,"threads_started":1}
I20260620 14:53:15.420974 23575 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:53:15.401481 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59386 (local address 127.0.97.60:38493)
0620 14:53:15.401726 (+ 245us) server_negotiation.cc:206] Beginning negotiation
0620 14:53:15.401736 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0620 14:53:15.412862 (+ 11126us) server_negotiation.cc:407] Connection header received
0620 14:53:15.413853 (+ 991us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:53:15.413862 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:53:15.413985 (+ 123us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:53:15.414053 (+ 68us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 14:53:15.414934 (+ 881us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:53:15.416049 (+ 1115us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:53:15.418508 (+ 2459us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:53:15.419149 (+ 641us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:53:15.419254 (+ 105us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:53:15.420104 (+ 850us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 14:53:15.420256 (+ 152us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0620 14:53:15.420346 (+ 90us) server_negotiation.cc:1035] Waiting for connection context
0620 14:53:15.420577 (+ 231us) server_negotiation.cc:299] Negotiation successful
0620 14:53:15.420669 (+ 92us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":82}
I20260620 14:53:15.421168 23569 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:53:15.412930 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54774 (local address 127.0.97.61:38589)
0620 14:53:15.413117 (+ 187us) server_negotiation.cc:206] Beginning negotiation
0620 14:53:15.413123 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 14:53:15.413773 (+ 650us) server_negotiation.cc:407] Connection header received
0620 14:53:15.414026 (+ 253us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:53:15.414032 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:53:15.414118 (+ 86us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:53:15.414178 (+ 60us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 14:53:15.415060 (+ 882us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:53:15.416180 (+ 1120us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:53:15.418555 (+ 2375us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:53:15.419225 (+ 670us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:53:15.419334 (+ 109us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:53:15.420276 (+ 942us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 14:53:15.420431 (+ 155us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0620 14:53:15.420498 (+ 67us) server_negotiation.cc:1035] Waiting for connection context
0620 14:53:15.420740 (+ 242us) server_negotiation.cc:299] Negotiation successful
0620 14:53:15.420840 (+ 100us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":75}
I20260620 14:53:15.422289 23581 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:53:15.401319 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58854 (local address 127.0.97.62:35275)
0620 14:53:15.413291 (+ 11972us) server_negotiation.cc:206] Beginning negotiation
0620 14:53:15.413300 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0620 14:53:15.413325 (+ 25us) server_negotiation.cc:407] Connection header received
0620 14:53:15.413475 (+ 150us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:53:15.413482 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:53:15.413717 (+ 235us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:53:15.413904 (+ 187us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 14:53:15.414835 (+ 931us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:53:15.416048 (+ 1213us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:53:15.418687 (+ 2639us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:53:15.419316 (+ 629us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:53:15.419441 (+ 125us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:53:15.420351 (+ 910us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 14:53:15.420860 (+ 509us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0620 14:53:15.420969 (+ 109us) server_negotiation.cc:1035] Waiting for connection context
0620 14:53:15.421471 (+ 502us) server_negotiation.cc:299] Negotiation successful
0620 14:53:15.421562 (+ 91us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":11800,"thread_start_us":11812,"threads_started":1}
I20260620 14:53:15.428923 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskUci3DU/build/asan/bin/kudu pid 23220
I20260620 14:53:15.430094 23282 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 14:53:15.544586 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskUci3DU/build/asan/bin/kudu with pid 23220
I20260620 14:53:15.574241 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskUci3DU/build/asan/bin/kudu pid 23294
I20260620 14:53:15.575701 23356 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 14:53:15.696496 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskUci3DU/build/asan/bin/kudu with pid 23294
I20260620 14:53:15.717235 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskUci3DU/build/asan/bin/kudu pid 23369
I20260620 14:53:15.718365 23431 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 14:53:15.829383 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskUci3DU/build/asan/bin/kudu with pid 23369
2026-06-20T14:53:15Z chronyd exiting
[ OK ] ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys/1 (5015 ms)
[----------] 2 tests from ConnectToFollowerMasterTest (9958 ms total)
[----------] Global test environment tear-down
[==========] 56 tests from 4 test suites ran. (366698 ms total)
[ PASSED ] 55 tests.
[ FAILED ] 1 test, listed below:
[ FAILED ] SecurityITest.TestJwtMiniCluster
1 FAILED TEST